Introduction

The bank data is anonimyzed, each row containing 20 values. Half of them are categorical and the rest numeric.

In the following we will explore the data, split train and test datasets, prepare it for a model, train a model and predict the target value for the test set. We will then try to identify the 5 most predictive values and we will use them in a new model to compare the benefits.

In [1]:
import datetime
import warnings
import numpy as np
import pandas as pd
import seaborn as sns
import xgboost as xgb
import matplotlib.pyplot as plt
import plotly.graph_objects as go

from sklearn.metrics import mean_squared_error, confusion_matrix
from sklearn.metrics import roc_auc_score, roc_curve, recall_score
from sklearn.model_selection import train_test_split, GridSearchCV
warnings.filterwarnings('ignore')

# run external scripts
%run ./plot_funcs.py

Load data

Load data and split into test and train.

In [2]:
%%time
df = pd.read_csv('data_set.csv', header=0)
cols = df.columns.to_list()[0].split(sep=';')
df = df[df.columns.to_list()[0]].str.split(';', expand=True)

for i, col in enumerate(cols):
    cols[i] = col.replace('"', '')

df.columns = cols

# set feature types with memory optimisation
data_type = {'age': np.int8, 'duration': np.int16, 'campaign': np.int8,
             'pdays': np.int16, 'previous': np.int8, 'emp.var.rate':
            np.float32, 'cons.price.idx': np.float32, 'cons.conf.idx':
            np.float32, 'euribor3m': np.float32, 'nr.employed': np.float32,
            'y': np.int8}

df['y'].replace(to_replace={'"yes"': 1, '"no"': 0}, inplace=True)
df = df.astype(dtype=data_type, errors='raise')
Wall time: 369 ms

Split train/test sample and categorical/numerical variables.

In [3]:
%%time
# replace 999 with a negative number -1
df['pdays'] = df['pdays'].where(df['pdays']!=999, -1)

# split train and test set 80-20%
train_df, test_df = train_test_split(df, test_size=0.2, shuffle=True)

categ_feat = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week',
              'poutcome']
numer_feat = ['age', 'duration', 'campaign', 'pdays', 'previous', 'emp.var.rate', 'cons.price.idx',
              'cons.conf.idx', 'euribor3m', 'nr.employed']
target = 'y'
Wall time: 18.4 ms

Data exploration

Check the data

Let's check the train and test set.

In [4]:
train_df.shape, test_df.shape
Out[4]:
((32950, 21), (8238, 21))

Train and test data have 32,950 and 8,238 entries respectivelly and 21 columns.

Let's glimpse train and test dataset.

In [5]:
train_df.head()
Out[5]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
5303 43 "blue-collar" "married" "basic.4y" "unknown" "yes" "no" "telephone" "may" "fri" ... 3 -1 0 "nonexistent" 1.1 93.994003 -36.400002 4.857 5191.000000 0
13826 35 "admin." "married" "high.school" "unknown" "no" "no" "telephone" "jul" "fri" ... 1 -1 0 "nonexistent" 1.4 93.917999 -42.700001 4.963 5228.100098 0
30138 60 "retired" "divorced" "professional.course" "no" "yes" "no" "cellular" "apr" "thu" ... 1 5 2 "failure" -1.8 93.074997 -47.099998 1.365 5099.100098 1
14975 32 "admin." "single" "basic.9y" "no" "no" "no" "cellular" "jul" "thu" ... 1 -1 0 "nonexistent" 1.4 93.917999 -42.700001 4.958 5228.100098 0
26413 38 "technician" "married" "university.degree" "no" "yes" "no" "cellular" "nov" "thu" ... 1 -1 0 "nonexistent" -0.1 93.199997 -42.000000 4.076 5195.799805 0

5 rows × 21 columns

In [6]:
test_df.head()
Out[6]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
1382 45 "admin." "married" "high.school" "no" "yes" "no" "telephone" "may" "thu" ... 4 -1 0 "nonexistent" 1.1 93.994003 -36.400002 4.855 5191.000000 0
13883 33 "blue-collar" "married" "basic.9y" "no" "no" "no" "cellular" "jul" "fri" ... 2 -1 0 "nonexistent" 1.4 93.917999 -42.700001 4.963 5228.100098 0
33743 43 "blue-collar" "married" "basic.4y" "no" "no" "yes" "cellular" "may" "wed" ... 1 -1 1 "failure" -1.8 92.892998 -46.200001 1.281 5099.100098 0
5523 28 "blue-collar" "married" "basic.6y" "no" "yes" "no" "telephone" "may" "mon" ... 2 -1 0 "nonexistent" 1.1 93.994003 -36.400002 4.857 5191.000000 0
40404 41 "entrepreneur" "married" "university.degree" "no" "yes" "no" "cellular" "aug" "thu" ... 1 -1 0 "nonexistent" -1.7 94.027000 -38.299999 0.904 4991.600098 1

5 rows × 21 columns

Datasets contain:

  • 'job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'poutcome' (string);
  • target 'y' (binary);
  • 'age', 'duration', 'campaign', 'pdays', 'previous', 'emp.var.rate', 'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed' numerical variables (floats and ints);

Let's check if there is any missing data. We will also chech the type of data (correctly assigned after exploring features' range of values and type).

In [7]:
%%time
missing_data(train_df)
Wall time: 85.8 ms
Out[7]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
Total 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
Percent 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
Types int8 object object object object object object object object object ... int8 int16 int8 object float32 float32 float32 float32 float32 int8

3 rows × 21 columns

Here we check test dataset.

In [8]:
%%time
missing_data(test_df)
Wall time: 18.4 ms
Out[8]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
Total 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
Percent 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
Types int8 object object object object object object object object object ... int8 int16 int8 object float32 float32 float32 float32 float32 int8

3 rows × 21 columns

There are no missing data in train and test datasets. Let's check the numerical values in train and test dataset.

In [9]:
%%time
train_df.describe()
Wall time: 20.8 ms
Out[9]:
age duration campaign pdays previous emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
count 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000 32950.000000
mean 40.046555 258.562853 2.566555 -0.735417 0.174810 0.084100 93.594437 -40.484409 3.624824 5167.729492 0.113323
std 10.439879 259.607865 2.764632 1.525998 0.498719 1.570388 0.579099 4.627327 1.733994 72.453598 0.316992
min 17.000000 0.000000 1.000000 -1.000000 0.000000 -3.400000 92.200996 -50.799999 0.634000 4963.600098 0.000000
25% 32.000000 103.000000 1.000000 -1.000000 0.000000 -1.800000 93.074997 -42.700001 1.344000 5099.100098 0.000000
50% 38.000000 180.000000 2.000000 -1.000000 0.000000 1.100000 93.797997 -41.799999 4.857000 5191.000000 0.000000
75% 47.000000 320.000000 3.000000 -1.000000 0.000000 1.400000 93.994003 -36.400002 4.961000 5228.100098 0.000000
max 98.000000 4918.000000 56.000000 25.000000 7.000000 1.400000 94.766998 -26.900000 5.045000 5228.100098 1.000000
In [10]:
%time
test_df.describe()
Wall time: 0 ns
Out[10]:
age duration campaign pdays previous emp.var.rate cons.price.idx cons.conf.idx euribor3m nr.employed y
count 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000 8238.000000
mean 39.934086 257.173707 2.571741 -0.768269 0.165574 0.073004 93.564903 -40.575821 3.607773 5167.306152 0.109978
std 10.346546 257.973384 2.791598 1.445738 0.479285 1.573305 0.578510 4.632864 1.736351 71.475037 0.312882
min 17.000000 1.000000 1.000000 -1.000000 0.000000 -3.400000 92.200996 -50.799999 0.634000 4963.600098 0.000000
25% 32.000000 101.000000 1.000000 -1.000000 0.000000 -1.800000 93.074997 -42.700001 1.344000 5099.100098 0.000000
50% 38.000000 178.000000 2.000000 -1.000000 0.000000 1.100000 93.444000 -41.799999 4.857000 5191.000000 0.000000
75% 47.000000 317.000000 3.000000 -1.000000 0.000000 1.400000 93.994003 -36.400002 4.961000 5228.100098 0.000000
max 92.000000 3785.000000 42.000000 27.000000 6.000000 1.400000 94.766998 -26.900000 5.045000 5228.100098 1.000000

We can make few observations here:

  • standard deviation is relatively large for both train and test variable data;
  • min, max, mean, std values for train and test data looks quite close;
  • mean values are distributed over a large range.

The number of values in train and test set is the same. Let's plot the scatter plot for train and test set for few of the features.

We will show just 25% of the train data amd all test data. On x axis we show train values and on the y axis we show the test values.

In [11]:
features = numer_feat

plot_feature_scatter(train_df[:len(test_df)], test_df, features)
<Figure size 432x288 with 0 Axes>

Let's check the distribution of target value in train dataset.

In [12]:
sns.countplot(train_df[target], palette='Set3')
Out[12]:
<matplotlib.axes._subplots.AxesSubplot at 0x1bdc0926808>
In [13]:
print("There are {0:.3f}% target values with 1".format(100 * train_df[target].value_counts()[1]/train_df.shape[0]))
There are 11.332% target values with 1

Let's check the distribution of target value in test dataset.

In [14]:
sns.countplot(test_df[target], palette='Set3')
Out[14]:
<matplotlib.axes._subplots.AxesSubplot at 0x1bdc0905ac8>
In [15]:
print("There are {0:.2f}% target values with 1".format(100 * test_df[target].value_counts()[1]/test_df.shape[0]))
There are 11.00% target values with 1

The data is unbalanced with respect with target value. For the shake of speeding up the analysis, we will assume that we did a good job into splitting the train and test sample and they are both representative of the overall sample.

With this in mind, we will dig deeper into the investigation of categorical values on the train test.

Job

In [16]:
train_df['job'].unique()
Out[16]:
array(['"blue-collar"', '"admin."', '"retired"', '"technician"',
       '"services"', '"management"', '"entrepreneur"', '"self-employed"',
       '"student"', '"unemployed"', '"unknown"', '"housemaid"'],
      dtype=object)
In [17]:
categ_feat_dfs = []
categ_feat_test_dfs = []

job_cat = pd.get_dummies(train_df['job'], prefix='job', drop_first=False)
job_cat.columns = [col.replace('"', '') for col in job_cat.columns]
job_cat.drop(['job_unknown'], axis=1, inplace=True)

job_cat_test = pd.get_dummies(test_df['job'], prefix='job', drop_first=False)
job_cat_test.columns = [col.replace('"', '') for col in job_cat_test.columns]
job_cat_test.drop(['job_unknown'], axis=1, inplace=True)

categ_feat_test_dfs.append(job_cat_test)
categ_feat_dfs.append(job_cat)
In [18]:
plot_feature_categ(train_df, 'job', target)
<Figure size 432x288 with 0 Axes>

Marital

In [19]:
train_df['marital'].unique()
Out[19]:
array(['"married"', '"divorced"', '"single"', '"unknown"'], dtype=object)
In [20]:
marital_cat = pd.get_dummies(train_df['marital'], prefix='marital', drop_first=False)
marital_cat.columns = [col.replace('"', '') for col in marital_cat.columns]
marital_cat.drop(['marital_unknown'], axis=1, inplace=True)

marital_cat_test = pd.get_dummies(test_df['marital'], prefix='marital', drop_first=False)
marital_cat_test.columns = [col.replace('"', '') for col in marital_cat_test.columns]
marital_cat_test.drop(['marital_unknown'], axis=1, inplace=True)

categ_feat_test_dfs.append(marital_cat_test)
categ_feat_dfs.append(marital_cat)
In [21]:
plot_feature_categ(train_df, 'marital', target)
<Figure size 432x288 with 0 Axes>

Education

In [22]:
train_df['education'].unique()
Out[22]:
array(['"basic.4y"', '"high.school"', '"professional.course"',
       '"basic.9y"', '"university.degree"', '"basic.6y"', '"unknown"',
       '"illiterate"'], dtype=object)
In [23]:
education_cat = pd.get_dummies(train_df['education'], prefix='education', drop_first=False)
education_cat.columns = [col.replace('"', '') for col in education_cat.columns]
education_cat.drop(['education_unknown'], axis=1, inplace=True)

education_cat_test = pd.get_dummies(test_df['education'], prefix='education', drop_first=False)
education_cat_test.columns = [col.replace('"', '') for col in education_cat_test.columns]
education_cat_test.drop(['education_unknown'], axis=1, inplace=True)

categ_feat_test_dfs.append(education_cat_test)
categ_feat_dfs.append(education_cat)
In [24]:
plot_feature_categ(train_df, 'education', target)
<Figure size 432x288 with 0 Axes>

Default

In [25]:
train_df['default'].unique()
Out[25]:
array(['"unknown"', '"no"', '"yes"'], dtype=object)
In [26]:
default_cat = pd.get_dummies(train_df['default'], prefix='default', drop_first=False)
default_cat.columns = [col.replace('"', '') for col in default_cat.columns]
try:
    default_cat.drop(['default_yes'], axis=1, inplace=True)
except:
    pass

default_cat_test = pd.get_dummies(test_df['default'], prefix='default', drop_first=False)
default_cat_test.columns = [col.replace('"', '') for col in default_cat_test.columns]
try:
    default_cat_test.drop(['default_yes'], axis=1, inplace=True)
except:
    pass

categ_feat_test_dfs.append(default_cat_test)
categ_feat_dfs.append(default_cat)
In [27]:
plot_feature_categ(train_df, 'default', target)
<Figure size 432x288 with 0 Axes>

Housing

In [28]:
train_df['housing'].unique()
Out[28]:
array(['"yes"', '"no"', '"unknown"'], dtype=object)
In [29]:
housing_cat = pd.get_dummies(train_df['housing'], prefix='housing', drop_first=False)
housing_cat.columns = [col.replace('"', '') for col in housing_cat.columns]
housing_cat.drop(['housing_unknown'], axis=1, inplace=True)

housing_cat_test = pd.get_dummies(test_df['housing'], prefix='housing', drop_first=False)
housing_cat_test.columns = [col.replace('"', '') for col in housing_cat_test.columns]
housing_cat_test.drop(['housing_unknown'], axis=1, inplace=True)

categ_feat_test_dfs.append(housing_cat_test)
categ_feat_dfs.append(housing_cat)
In [30]:
plot_feature_categ(train_df, 'housing', target)
<Figure size 432x288 with 0 Axes>

Loan

In [31]:
train_df['loan'].unique()
Out[31]:
array(['"no"', '"yes"', '"unknown"'], dtype=object)
In [32]:
loan_cat = pd.get_dummies(train_df['loan'], prefix='loan', drop_first=False)
loan_cat.columns = [col.replace('"', '') for col in loan_cat.columns]
loan_cat.drop(['loan_unknown'], axis=1, inplace=True)

loan_cat_test = pd.get_dummies(test_df['loan'], prefix='loan', drop_first=False)
loan_cat_test.columns = [col.replace('"', '') for col in loan_cat_test.columns]
loan_cat_test.drop(['loan_unknown'], axis=1, inplace=True)

categ_feat_test_dfs.append(loan_cat_test)
categ_feat_dfs.append(loan_cat)
In [33]:
plot_feature_categ(train_df, 'loan', target)
<Figure size 432x288 with 0 Axes>

Contact

In [34]:
train_df['contact'].unique()
Out[34]:
array(['"telephone"', '"cellular"'], dtype=object)
In [35]:
contact_cat = pd.get_dummies(train_df['contact'], prefix='contact', drop_first=False)
contact_cat.columns = [col.replace('"', '') for col in contact_cat.columns]
contact_cat.drop(['contact_telephone'], axis=1, inplace=True)

contact_cat_test = pd.get_dummies(test_df['contact'], prefix='contact', drop_first=False)
contact_cat_test.columns = [col.replace('"', '') for col in contact_cat_test.columns]
contact_cat_test.drop(['contact_telephone'], axis=1, inplace=True)

categ_feat_test_dfs.append(contact_cat_test)
categ_feat_dfs.append(contact_cat)
In [36]:
plot_feature_categ(train_df, 'contact', target)
<Figure size 432x288 with 0 Axes>

Month

In [37]:
train_df['month'].unique()
Out[37]:
array(['"may"', '"jul"', '"apr"', '"nov"', '"jun"', '"aug"', '"sep"',
       '"oct"', '"dec"', '"mar"'], dtype=object)
In [38]:
month_cat = pd.get_dummies(train_df['month'], prefix='month', drop_first=False)
month_cat.columns = [col.replace('"', '') for col in month_cat.columns]
month_cat.drop(['month_mar'], axis=1, inplace=True)

month_cat_test = pd.get_dummies(test_df['month'], prefix='month', drop_first=False)
month_cat_test.columns = [col.replace('"', '') for col in month_cat_test.columns]
month_cat_test.drop(['month_mar'], axis=1, inplace=True)

categ_feat_test_dfs.append(month_cat_test)
categ_feat_dfs.append(month_cat)
In [39]:
order = ['"mar"', '"apr"', '"may"', '"jun"', '"jul"', '"aug"', '"sep"', '"oct"', '"nov"', '"dec"']
plot_feature_categ(train_df, 'month', target, order=order)
<Figure size 432x288 with 0 Axes>

Day of week

In [40]:
train_df['day_of_week'].unique()
Out[40]:
array(['"fri"', '"thu"', '"tue"', '"mon"', '"wed"'], dtype=object)
In [41]:
day_cat = pd.get_dummies(train_df['day_of_week'], prefix='day', drop_first=False)
day_cat.columns = [col.replace('"', '') for col in day_cat.columns]
day_cat.drop(['day_fri'], axis=1, inplace=True)

day_cat_test = pd.get_dummies(test_df['day_of_week'], prefix='day', drop_first=False)
day_cat_test.columns = [col.replace('"', '') for col in day_cat_test.columns]
day_cat_test.drop(['day_fri'], axis=1, inplace=True)

categ_feat_test_dfs.append(day_cat_test)
categ_feat_dfs.append(day_cat)
In [42]:
order = ['"mon"', '"tue"', '"wed"', '"thu"', '"fri"']
plot_feature_categ(train_df, 'day_of_week', target, order=order)
<Figure size 432x288 with 0 Axes>

Poutcome

In [43]:
train_df['poutcome'].unique()
Out[43]:
array(['"nonexistent"', '"failure"', '"success"'], dtype=object)
In [44]:
poutcome_cat = pd.get_dummies(train_df['poutcome'], prefix='poutcome', drop_first=False)
poutcome_cat.columns = [col.replace('"', '') for col in poutcome_cat.columns]
poutcome_cat.drop(['poutcome_nonexistent'], axis=1, inplace=True)

poutcome_cat_test = pd.get_dummies(test_df['poutcome'], prefix='poutcome', drop_first=False)
poutcome_cat_test.columns = [col.replace('"', '') for col in poutcome_cat_test.columns]
poutcome_cat_test.drop(['poutcome_nonexistent'], axis=1, inplace=True)

categ_feat_test_dfs.append(poutcome_cat_test)
categ_feat_dfs.append(poutcome_cat)
In [45]:
plot_feature_categ(train_df, 'poutcome', target)
<Figure size 432x288 with 0 Axes>

Density plots of features

Let's show now the density plot of variables in train dataset (numerical features).

We represent with different colors the distribution for values with target value 0 and 1.

The first 10 numerical values are displayed in the following cell.

In [46]:
t0 = train_df.loc[train_df[target] == 0]
t1 = train_df.loc[train_df[target] == 1]
plot_feature_distribution(t0, t1, '0', '1', numer_feat)
<Figure size 432x288 with 0 Axes>

We can observe that there is a considerable number of features with significant different distribution for the two target values.

For example, duration, pdays, previous, emp.var.rate, cons.price.idx and nr.employed.

Also some features, like emp.var.rate, cons.price.idx, cons.conf.idx, var_55 and euribor3m show a distribution that resambles to a multivariate distribution. This also is an early indicator that a tree-based model can be very performant.

We will take this into consideration in the future for the selection of the features for our prediction model.

Le't s now look to the distribution of the same features in parallel in train and test datasets.

The first 10 values are displayed in the following cell.

In [47]:
plot_feature_distribution(train_df, test_df, 'train', 'test', numer_feat)
<Figure size 432x288 with 0 Axes>

The train and test seems not to be very well ballanced with respect with distribution of the numeric variables, something that could affect the performance of the final model.

Distribution of mean and std

Let's check the distribution of the mean values (numerical) per row in the train and test set.

In [48]:
plt.figure(figsize=(16,6))
features = train_df.columns.values[:-1]
plt.title("Distribution of mean values per row in the train and test set")
sns.distplot(train_df[numer_feat].mean(axis=1),color="green", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].mean(axis=1),color="blue", kde=True,bins=50, label='test')
plt.legend()
plt.show()

Let's check the distribution of the mean values per columns in the train and test set.

In [49]:
plt.figure(figsize=(16,6))
plt.title("Distribution of mean values per column in the train and test set")
sns.distplot(train_df[numer_feat].mean(axis=0),color="magenta",kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].mean(axis=0),color="darkblue", kde=True,bins=50, label='test')
plt.legend()
plt.show()

Let's show the distribution of standard deviation of values per row for train and test datasets.

In [50]:
plt.figure(figsize=(16,6))
plt.title("Distribution of std values per row in the train and test set")
sns.distplot(train_df[numer_feat].std(axis=1),color="black", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].std(axis=1),color="red", kde=True,bins=50, label='test')
plt.legend();plt.show()

Let's check the distribution of the standard deviation of values per columns in the train and test datasets.

In [51]:
plt.figure(figsize=(16,6))
plt.title("Distribution of std values per column in the train and test set")
sns.distplot(train_df[numer_feat].std(axis=0),color="blue",kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].std(axis=0),color="green", kde=True,bins=50, label='test')
plt.legend(); plt.show()

Let's check now the distribution of the mean value per row in the train dataset, grouped by value of target.

In [52]:
t0 = train_df.loc[train_df[target] == 0]
t1 = train_df.loc[train_df[target] == 1]
plt.figure(figsize=(16,6))
plt.title("Distribution of mean values per row in the train set")
sns.distplot(t0[numer_feat].mean(axis=1),color="red", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].mean(axis=1),color="blue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Let's check now the distribution of the mean value per column in the train dataset, grouped by value of target.

In [53]:
plt.figure(figsize=(16,6))
plt.title("Distribution of mean values per column in the train set")
sns.distplot(t0[numer_feat].mean(axis=0),color="green", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].mean(axis=0),color="darkblue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Distribution of min and max

Let's check the distribution of min per row in the train and test set.

In [54]:
plt.figure(figsize=(16,6))
plt.title("Distribution of min values per row in the train and test set")
sns.distplot(train_df[numer_feat].min(axis=1),color="red", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].min(axis=1),color="orange", kde=True,bins=50, label='test')
plt.legend()
plt.show()

A long variation that centres around clusters is observed.

Let's now show the distribution of min per column in the train and test set.

In [55]:
plt.figure(figsize=(16,6))
plt.title("Distribution of min values per column in the train and test set")
sns.distplot(train_df[numer_feat].min(axis=0),color="magenta", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].min(axis=0),color="darkblue", kde=True,bins=50, label='test')
plt.legend()
plt.show()

Yet again, one (or very few) columns has very big values, compared to the rest (majority) of the columns.

Let's check now the distribution of max values per rows for train and test set.

In [56]:
plt.figure(figsize=(16,6))
plt.title("Distribution of max values per row in the train and test set")
sns.distplot(train_df[numer_feat].max(axis=1),color="brown", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].max(axis=1),color="yellow", kde=True,bins=50, label='test')
plt.legend()
plt.show()

Let's show now the max distribution on columns for train and test set.

In [57]:
plt.figure(figsize=(16,6))
plt.title("Distribution of max values per column in the train and test set")
sns.distplot(train_df[numer_feat].max(axis=0),color="blue", kde=True,bins=50, label='train')
sns.distplot(test_df[numer_feat].max(axis=0),color="red", kde=True,bins=50, label='test')
plt.legend()
plt.show()

Let's show now the distributions of min values per row in train set, separated on the values of target (0 and 1).

In [58]:
t0 = train_df.loc[train_df[target] == 0]
t1 = train_df.loc[train_df[target] == 1]
plt.figure(figsize=(16,6))
plt.title("Distribution of min values per row in the train set")
sns.distplot(t0[numer_feat].min(axis=1),color="orange", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].min(axis=1),color="darkblue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

We show here the distribution of min values per columns in train set.

In [59]:
plt.figure(figsize=(16,6))
plt.title("Distribution of min values per column in the train set")
sns.distplot(t0[numer_feat].min(axis=0),color="red", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].min(axis=0),color="blue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Let's show now the distribution of max values per row in the train set.

In [60]:
plt.figure(figsize=(16,6))
plt.title("Distribution of max values per row in the train set")
sns.distplot(t0[numer_feat].max(axis=1),color="gold", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].max(axis=1),color="darkblue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Let's show also the distribution of max values per columns in the train set.

In [61]:
plt.figure(figsize=(16,6))
plt.title("Distribution of max values per column in the train set")
sns.distplot(t0[numer_feat].max(axis=0),color="red", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].max(axis=0),color="blue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Distribution of skew and kurtosis

Let's see now what is the distribution of skew values per rows and columns.

Let's see first the distribution of skewness calculated per rows in train and test sets.

In [62]:
plt.figure(figsize=(16,6))
plt.title("Distribution of skew per row in the train and test set")
sns.distplot(train_df[numer_feat].skew(axis=1),color="red", kde=True,bins=20, label='train')
sns.distplot(test_df[numer_feat].skew(axis=1),color="orange", kde=True,bins=20, label='test')
plt.legend()
plt.show()

Let's see first the distribution of skewness calculated per columns in train and test set.

In [63]:
plt.figure(figsize=(16,6))
plt.title("Distribution of skew per column in the train and test set")
sns.distplot(train_df[numer_feat].skew(axis=0),color="magenta", kde=True,bins=30, label='train')
sns.distplot(test_df[numer_feat].skew(axis=0),color="darkblue", kde=True,bins=30, label='test')
plt.legend()
plt.show()

Let's see now what is the distribution of kurtosis values per rows and columns.

Let's see first the distribution of kurtosis calculated per rows in train and test sets.

In [64]:
plt.figure(figsize=(16,6))
plt.title("Distribution of kurtosis per row in the train and test set")
sns.distplot(train_df[numer_feat].kurtosis(axis=1),color="darkblue", kde=True,bins=20, label='train')
sns.distplot(test_df[numer_feat].kurtosis(axis=1),color="yellow", kde=True,bins=20, label='test')
plt.legend()
plt.show()

Let's see first the distribution of kurtosis calculated per columns in train and test sets.

In [65]:
plt.figure(figsize=(16,6))
plt.title("Distribution of kurtosis per column in the train and test set")
sns.distplot(train_df[numer_feat].kurtosis(axis=0),color="magenta", kde=True,bins=40, label='train')
sns.distplot(test_df[numer_feat].kurtosis(axis=0),color="green", kde=True,bins=40, label='test')
plt.legend()
plt.show()

Let's see now the distribution of skewness on rows in train separated for values of target 0 and 1.

In [66]:
t0 = train_df.loc[train_df[target] == 0]
t1 = train_df.loc[train_df[target] == 1]
plt.figure(figsize=(16,6))
plt.title("Distribution of skew values per row in the train set")
sns.distplot(t0[numer_feat].skew(axis=1),color="red", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].skew(axis=1),color="blue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Let's see now the distribution of skewness on columns in train separated for values of target 0 and 1.

In [67]:
plt.figure(figsize=(16,6))
plt.title("Distribution of skew values per column in the train set")
sns.distplot(t0[numer_feat].skew(axis=0),color="red", kde=True,bins=30, label='target = 0')
sns.distplot(t1[numer_feat].skew(axis=0),color="blue", kde=True,bins=30, label='target = 1')
plt.legend(); plt.show()

Let's see now the distribution of kurtosis on rows in train separated for values of target 0 and 1.

In [68]:
plt.figure(figsize=(16,6))
plt.title("Distribution of kurtosis values per row in the train set")
sns.distplot(t0[numer_feat].kurtosis(axis=1),color="red", kde=True,bins=50, label='target = 0')
sns.distplot(t1[numer_feat].kurtosis(axis=1),color="blue", kde=True,bins=50, label='target = 1')
plt.legend(); plt.show()

Let's see now the distribution of kurtosis on columns in train separated for values of target 0 and 1.

In [69]:
plt.figure(figsize=(16,6))
plt.title("Distribution of kurtosis values per column in the train set")
sns.distplot(t0[numer_feat].kurtosis(axis=0),color="red", kde=True,bins=10, label='target = 0')
sns.distplot(t1[numer_feat].kurtosis(axis=0),color="blue", kde=True,bins=10, label='target = 1')
plt.legend(); plt.show()

Features correlation (including categorical)

We calculate now the correlations between the features in train set.
The following table shows the first 10 least correlated features.

In [70]:
%%time
correlations = train_df[numer_feat].corr().abs().unstack().sort_values(kind="quicksort").reset_index()
correlations = correlations[correlations['level_0'] != correlations['level_1']]
correlations.head(10)
Wall time: 11.4 ms
Out[70]:
level_0 level_1 0
0 age duration 0.000960
1 duration age 0.000960
2 cons.price.idx age 0.001304
3 age cons.price.idx 0.001304
4 emp.var.rate age 0.003197
5 age emp.var.rate 0.003197
6 campaign age 0.006921
7 age campaign 0.006921
8 cons.conf.idx duration 0.007287
9 duration cons.conf.idx 0.007287

Let's look to the top most correlated features, besides the same feature pairs.

In [71]:
correlations.tail(10)
Out[71]:
level_0 level_1 0
80 cons.price.idx euribor3m 0.686035
81 euribor3m cons.price.idx 0.686035
82 cons.price.idx emp.var.rate 0.774582
83 emp.var.rate cons.price.idx 0.774582
84 emp.var.rate nr.employed 0.906408
85 nr.employed emp.var.rate 0.906408
86 nr.employed euribor3m 0.945150
87 euribor3m nr.employed 0.945150
88 emp.var.rate euribor3m 0.971962
89 euribor3m emp.var.rate 0.971962

Let's look to the top most correlated features, besides the same feature pairs, including the categorical features.

In [72]:
%%time
dfs_for_concat = [train_df.drop(columns=categ_feat, axis=1)]
dfs_for_concat = dfs_for_concat + categ_feat_dfs
train_df_total = pd.concat(dfs_for_concat, axis=1, join='outer')
train_df_total = train_df_total.drop(columns=['y'], axis=1)
train_df_total['y'] = train_df['y']

correlations2 = train_df_total[numer_feat].corr().abs().unstack().sort_values(kind="quicksort").reset_index()
correlations2 = correlations2[correlations2['level_0'] != correlations2['level_1']]
correlations2.head(10)
Wall time: 59 ms
Out[72]:
level_0 level_1 0
0 age duration 0.000960
1 duration age 0.000960
2 cons.price.idx age 0.001304
3 age cons.price.idx 0.001304
4 emp.var.rate age 0.003197
5 age emp.var.rate 0.003197
6 campaign age 0.006921
7 age campaign 0.006921
8 cons.conf.idx duration 0.007287
9 duration cons.conf.idx 0.007287

Let's look to the top most correlated features, besides the same feature pairs, including the categorical features.

In [73]:
correlations2.tail(10)
Out[73]:
level_0 level_1 0
80 cons.price.idx euribor3m 0.686035
81 euribor3m cons.price.idx 0.686035
82 cons.price.idx emp.var.rate 0.774582
83 emp.var.rate cons.price.idx 0.774582
84 emp.var.rate nr.employed 0.906408
85 nr.employed emp.var.rate 0.906408
86 nr.employed euribor3m 0.945150
87 euribor3m nr.employed 0.945150
88 emp.var.rate euribor3m 0.971962
89 euribor3m emp.var.rate 0.971962
In [74]:
# creating test_df_total
dfs_for_concat = [test_df.drop(columns=categ_feat, axis=1)]
dfs_for_concat = dfs_for_concat + categ_feat_test_dfs
test_df_total = pd.concat(dfs_for_concat, axis=1, join='outer')
test_df_total = test_df_total.drop(columns=['y'], axis=1)
test_df_total['y'] = test_df['y']

Visualise correlations on Train Dataset:

In [75]:
corr_heatmap_plot(test_df, cmap='coolwarm')

Visualise correlations on Test Dataset:

In [76]:
corr_heatmap_plot(train_df, cmap='coolwarm')

The correlation between some of the features is very high. Age is particularly not very correlated with other variables.

Duplicate values

Let's now check how many duplicate values exists per columns.

In [77]:
%%time
unique_max_train = []
unique_max_test = []
for feature in numer_feat:
    values = train_df[feature].value_counts()
    unique_max_train.append([feature, values.max(), values.idxmax()])
    values = test_df[feature].value_counts()
    unique_max_test.append([feature, values.max(), values.idxmax()])
Wall time: 13.9 ms

Let's show the top 15 max of duplicate values per train set.

In [78]:
np.transpose((pd.DataFrame(unique_max_train, columns=['Feature', 'Max duplicates', 'Value'])).\
            sort_values(by = 'Max duplicates', ascending=False).head(15))
Out[78]:
3 4 2 5 9 6 7 8 0 1
Feature pdays previous campaign emp.var.rate nr.employed cons.price.idx cons.conf.idx euribor3m age duration
Max duplicates 31704 28422 14136 12977 12977 6232 6232 2297 1559 143
Value -1 0 1 1.4 5228.1 93.994 -36.4 4.857 31 85

Let's see also the top 15 number of duplicates values per test set.

In [79]:
np.transpose((pd.DataFrame(unique_max_test, columns=['Feature', 'Max duplicates', 'Value'])).\
            sort_values(by = 'Max duplicates', ascending=False).head(15))
Out[79]:
3 4 2 5 9 6 7 8 0 1
Feature pdays previous campaign emp.var.rate nr.employed cons.price.idx cons.conf.idx euribor3m age duration
Max duplicates 7969 7141 3506 3257 3257 1531 1531 571 388 41
Value -1 0 1 1.4 5228.1 93.994 -36.4 4.857 31 136

Same columns in train and test set have the same or very close number of duplicates of same or very close values. This means that our test sample is of same quality as the training sample.

Feature engineering

Let's calculate for starting few aggregated values for the existing features.

In [80]:
%%time
idx = numer_feat
for dfn in [test_df_total, train_df_total]:
    dfn['sum'] = dfn[idx].sum(axis=1)  
    dfn['min'] = dfn[idx].min(axis=1)
    dfn['max'] = dfn[idx].max(axis=1)
    dfn['mean'] = dfn[idx].mean(axis=1)
    dfn['std'] = dfn[idx].std(axis=1)
    dfn['skew'] = dfn[idx].skew(axis=1)
    dfn['kurt'] = dfn[idx].kurtosis(axis=1)
    dfn['med'] = dfn[idx].median(axis=1)
Wall time: 56 ms

Let's check the new created features for the numeric values.

In [81]:
train_df_total[train_df_total.columns[-8:]].head()
Out[81]:
sum min max mean std skew kurt med
5303 5644.550781 -36.400002 5191.000000 564.455078 1629.296143 3.136243 9.874394 3.9285
13826 5509.681152 -42.700001 5228.100098 550.968140 1644.687988 3.152943 9.955990 3.1815
30138 6180.640137 -47.099998 5099.100098 618.064026 1603.202759 2.974046 9.017812 3.5000
14975 5502.676270 -42.700001 5228.100098 550.267639 1644.891357 3.153247 9.957438 3.1790
26413 5370.975586 -42.000000 5195.799805 537.097534 1637.417725 3.158549 9.982629 2.5380
In [82]:
test_df_total[test_df_total.columns[-8:]].head()
Out[82]:
sum min max mean std skew kurt med
1382 5405.548828 -36.400002 5191.000000 540.554871 1634.602417 3.157928 9.979708 4.4275
13883 5758.681152 -42.700001 5228.100098 575.868103 1640.503784 3.121633 9.801357 3.4815
33743 5258.273926 -46.200001 5099.100098 525.827393 1607.387695 3.158574 9.982761 1.1405
5523 5403.550781 -36.400002 5191.000000 540.355103 1634.761963 3.157292 9.976680 3.4285
40404 5823.531250 -38.299999 4991.600098 582.353149 1566.255737 3.044472 9.400020 0.9520
In [83]:
# swap target column to the end
train_df_total = train_df_total.drop(columns=['y'], axis=1)
train_df_total['y'] = train_df['y']
test_df_total = test_df_total.drop(columns=['y'], axis=1)
test_df_total['y'] = test_df['y']

Let's check the distribution of these new, engineered features.

We plot first the distribution of new features, grouped by value of corresponding target values.

In [84]:
t0 = train_df_total.loc[train_df_total[target] == 0]
t1 = train_df_total.loc[train_df_total[target] == 1]
features = train_df_total.columns[-9:-1] # 8 new features

plot_new_feature_distribution(t0, t1, 'target: 0', 'target: 1', features)
<Figure size 432x288 with 0 Axes>

Let's show the distribution of new features values for train and test.

In [85]:
plot_new_feature_distribution(train_df_total, test_df_total, 'train', 'test', features)
<Figure size 432x288 with 0 Axes>

We can see that the train_total and test_total have no difference in their distributions of the new features, which is good, while there is a big difference between some of the features between the two classes, like min, sum and median.

Let's check how many features we have now (should be 53 + 8 + 1 target).

In [86]:
print('Train and test columns: {} {}'.format(len(train_df_total.columns), len(test_df_total.columns)))
Train and test columns: 62 62

Model

From the train columns list, we drop the target to form the features list.

In [87]:
features = [c for c in train_df_total.columns if c!=target]
target = train_df_total[target]

We define the hyperparameters for the model.

In [88]:
ratio = len(t0)/len(t1) # ratio between the two imbalanced classes

parameters = {
    'booster': ['gbtree'],
    'verbosity': [0],
    'max_depth': [3, 5, 7,],
    'min_child_weight': [3, 5, 7],
    'gamma': [0.0, 0.15, 0.3],
    'subsample': [0.5, 1.0],
    'objective': ['binary:logistic'],
    'disable_default_eval_metric': [1],
    'eval_metric': ['auc'],
    'n_estimators': [1000],
    'scale_pos_weight': [ratio],
    'n_jobs': [8]
}

We now run GridSearchCV to identify the best hyperparameters.

In [89]:
grid_search = GridSearchCV(estimator=xgb.XGBClassifier(), param_grid=parameters, cv=10)
In [90]:
%%time
# perform grid search
grid_search.fit(train_df_total[features], target)
Wall time: 2h 33min 7s
Out[90]:
GridSearchCV(cv=10, error_score=nan,
             estimator=XGBClassifier(base_score=0.5, booster='gbtree',
                                     colsample_bylevel=1, colsample_bynode=1,
                                     colsample_bytree=1, gamma=0,
                                     learning_rate=0.1, max_delta_step=0,
                                     max_depth=3, min_child_weight=1,
                                     missing=None, n_estimators=100, n_jobs=1,
                                     nthread=None, objective='binary:logistic',
                                     random_state=0, reg_alpha=0, reg_lambda=1,
                                     scale_p...
                         'disable_default_eval_metric': [1],
                         'eval_metric': ['auc'], 'gamma': [0.0, 0.15, 0.3],
                         'max_depth': [3, 5, 7], 'min_child_weight': [3, 5, 7],
                         'n_estimators': [1000], 'n_jobs': [8],
                         'objective': ['binary:logistic'],
                         'scale_pos_weight': [7.8243170862346005],
                         'subsample': [0.5, 1.0], 'verbosity': [0]},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring=None, verbose=0)
In [91]:
# get best parameters
best_estimator = grid_search.best_estimator_
best_accuracy = grid_search.best_score_
best_parameters = grid_search.best_params_

# replace parameters
# best_parameters['verbosity'] = 1
best_parameters['verbose_eval'] = 100

print("Best Estimator:", best_estimator)
print("Best Accuracy:", best_accuracy)
print("Best Parameters:", best_parameters)
print("Balance Ratio:", round(ratio, 3))
Best Estimator: XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1,
              disable_default_eval_metric=1, eval_metric='auc', gamma=0.15,
              learning_rate=0.1, max_delta_step=0, max_depth=7,
              min_child_weight=3, missing=None, n_estimators=1000, n_jobs=8,
              nthread=None, objective='binary:logistic', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=7.8243170862346005,
              seed=None, silent=None, subsample=0.5, verbosity=0)
Best Accuracy: 0.9074962063732928
Best Parameters: {'booster': 'gbtree', 'disable_default_eval_metric': 1, 'eval_metric': 'auc', 'gamma': 0.15, 'max_depth': 7, 'min_child_weight': 3, 'n_estimators': 1000, 'n_jobs': 8, 'objective': 'binary:logistic', 'scale_pos_weight': 7.8243170862346005, 'subsample': 0.5, 'verbosity': 0, 'verbose_eval': 100}
Balance Ratio: 7.824

We run the 1st cross-validated model and test the performance on the test set.

Model 1 - With engineered features

In [92]:
%%time
clf1 = xgb.XGBClassifier(**best_parameters)
tr_val = (train_df_total[features], target)
ts_val = (test_df_total[features], test_df_total['y'])

clf1.fit(train_df_total[features], target, eval_metric='auc', eval_set=[tr_val, ts_val])
[0]	validation_0-auc:0.943033	validation_1-auc:0.931757
[1]	validation_0-auc:0.947304	validation_1-auc:0.934507
[2]	validation_0-auc:0.950168	validation_1-auc:0.936386
[3]	validation_0-auc:0.951532	validation_1-auc:0.937866
[4]	validation_0-auc:0.952819	validation_1-auc:0.939659
[5]	validation_0-auc:0.953941	validation_1-auc:0.939982
[6]	validation_0-auc:0.954973	validation_1-auc:0.940681
[7]	validation_0-auc:0.955155	validation_1-auc:0.941177
[8]	validation_0-auc:0.956056	validation_1-auc:0.942698
[9]	validation_0-auc:0.956704	validation_1-auc:0.943647
[10]	validation_0-auc:0.957483	validation_1-auc:0.943482
[11]	validation_0-auc:0.958116	validation_1-auc:0.943555
[12]	validation_0-auc:0.958502	validation_1-auc:0.943871
[13]	validation_0-auc:0.959032	validation_1-auc:0.943995
[14]	validation_0-auc:0.959517	validation_1-auc:0.944239
[15]	validation_0-auc:0.959986	validation_1-auc:0.944518
[16]	validation_0-auc:0.960691	validation_1-auc:0.944933
[17]	validation_0-auc:0.960889	validation_1-auc:0.944526
[18]	validation_0-auc:0.961562	validation_1-auc:0.944607
[19]	validation_0-auc:0.962011	validation_1-auc:0.944699
[20]	validation_0-auc:0.962294	validation_1-auc:0.94463
[21]	validation_0-auc:0.962383	validation_1-auc:0.944656
[22]	validation_0-auc:0.962731	validation_1-auc:0.94485
[23]	validation_0-auc:0.963048	validation_1-auc:0.945191
[24]	validation_0-auc:0.963453	validation_1-auc:0.945328
[25]	validation_0-auc:0.963632	validation_1-auc:0.945506
[26]	validation_0-auc:0.963719	validation_1-auc:0.945446
[27]	validation_0-auc:0.963964	validation_1-auc:0.945544
[28]	validation_0-auc:0.964239	validation_1-auc:0.945727
[29]	validation_0-auc:0.964515	validation_1-auc:0.945734
[30]	validation_0-auc:0.964545	validation_1-auc:0.945664
[31]	validation_0-auc:0.96467	validation_1-auc:0.945689
[32]	validation_0-auc:0.965009	validation_1-auc:0.945793
[33]	validation_0-auc:0.965156	validation_1-auc:0.945821
[34]	validation_0-auc:0.96555	validation_1-auc:0.946102
[35]	validation_0-auc:0.965646	validation_1-auc:0.946083
[36]	validation_0-auc:0.965903	validation_1-auc:0.945984
[37]	validation_0-auc:0.966044	validation_1-auc:0.945971
[38]	validation_0-auc:0.966172	validation_1-auc:0.946136
[39]	validation_0-auc:0.966534	validation_1-auc:0.946289
[40]	validation_0-auc:0.966726	validation_1-auc:0.946215
[41]	validation_0-auc:0.966919	validation_1-auc:0.946378
[42]	validation_0-auc:0.967058	validation_1-auc:0.946382
[43]	validation_0-auc:0.967277	validation_1-auc:0.94647
[44]	validation_0-auc:0.967619	validation_1-auc:0.946381
[45]	validation_0-auc:0.967809	validation_1-auc:0.946396
[46]	validation_0-auc:0.967926	validation_1-auc:0.946434
[47]	validation_0-auc:0.968154	validation_1-auc:0.946418
[48]	validation_0-auc:0.968379	validation_1-auc:0.94639
[49]	validation_0-auc:0.968653	validation_1-auc:0.946518
[50]	validation_0-auc:0.968936	validation_1-auc:0.946488
[51]	validation_0-auc:0.969065	validation_1-auc:0.946515
[52]	validation_0-auc:0.969146	validation_1-auc:0.946425
[53]	validation_0-auc:0.969454	validation_1-auc:0.946486
[54]	validation_0-auc:0.969575	validation_1-auc:0.946539
[55]	validation_0-auc:0.969868	validation_1-auc:0.946653
[56]	validation_0-auc:0.970022	validation_1-auc:0.946676
[57]	validation_0-auc:0.970085	validation_1-auc:0.94671
[58]	validation_0-auc:0.97026	validation_1-auc:0.946657
[59]	validation_0-auc:0.970377	validation_1-auc:0.946588
[60]	validation_0-auc:0.970548	validation_1-auc:0.946466
[61]	validation_0-auc:0.970746	validation_1-auc:0.946456
[62]	validation_0-auc:0.971119	validation_1-auc:0.946361
[63]	validation_0-auc:0.971258	validation_1-auc:0.946304
[64]	validation_0-auc:0.971421	validation_1-auc:0.9462
[65]	validation_0-auc:0.971651	validation_1-auc:0.946045
[66]	validation_0-auc:0.971991	validation_1-auc:0.946229
[67]	validation_0-auc:0.972268	validation_1-auc:0.94627
[68]	validation_0-auc:0.972357	validation_1-auc:0.946246
[69]	validation_0-auc:0.972422	validation_1-auc:0.946205
[70]	validation_0-auc:0.972569	validation_1-auc:0.946293
[71]	validation_0-auc:0.972679	validation_1-auc:0.946365
[72]	validation_0-auc:0.972869	validation_1-auc:0.946348
[73]	validation_0-auc:0.972995	validation_1-auc:0.946372
[74]	validation_0-auc:0.973098	validation_1-auc:0.946401
[75]	validation_0-auc:0.973282	validation_1-auc:0.946437
[76]	validation_0-auc:0.973543	validation_1-auc:0.946529
[77]	validation_0-auc:0.97363	validation_1-auc:0.946499
[78]	validation_0-auc:0.973825	validation_1-auc:0.946406
[79]	validation_0-auc:0.974083	validation_1-auc:0.946173
[80]	validation_0-auc:0.974436	validation_1-auc:0.945999
[81]	validation_0-auc:0.974638	validation_1-auc:0.945988
[82]	validation_0-auc:0.974912	validation_1-auc:0.94602
[83]	validation_0-auc:0.975062	validation_1-auc:0.945942
[84]	validation_0-auc:0.975125	validation_1-auc:0.945972
[85]	validation_0-auc:0.975459	validation_1-auc:0.945916
[86]	validation_0-auc:0.975601	validation_1-auc:0.945845
[87]	validation_0-auc:0.975825	validation_1-auc:0.945805
[88]	validation_0-auc:0.975972	validation_1-auc:0.945749
[89]	validation_0-auc:0.976214	validation_1-auc:0.945555
[90]	validation_0-auc:0.976377	validation_1-auc:0.945409
[91]	validation_0-auc:0.976565	validation_1-auc:0.945295
[92]	validation_0-auc:0.976701	validation_1-auc:0.945256
[93]	validation_0-auc:0.976934	validation_1-auc:0.945396
[94]	validation_0-auc:0.977112	validation_1-auc:0.945293
[95]	validation_0-auc:0.977282	validation_1-auc:0.945436
[96]	validation_0-auc:0.977434	validation_1-auc:0.945501
[97]	validation_0-auc:0.977533	validation_1-auc:0.945485
[98]	validation_0-auc:0.977591	validation_1-auc:0.945478
[99]	validation_0-auc:0.977777	validation_1-auc:0.945413
[100]	validation_0-auc:0.977882	validation_1-auc:0.945422
[101]	validation_0-auc:0.978033	validation_1-auc:0.945475
[102]	validation_0-auc:0.978215	validation_1-auc:0.945339
[103]	validation_0-auc:0.978358	validation_1-auc:0.945287
[104]	validation_0-auc:0.978535	validation_1-auc:0.945273
[105]	validation_0-auc:0.978662	validation_1-auc:0.9452
[106]	validation_0-auc:0.978815	validation_1-auc:0.945194
[107]	validation_0-auc:0.978895	validation_1-auc:0.945202
[108]	validation_0-auc:0.978955	validation_1-auc:0.945172
[109]	validation_0-auc:0.979169	validation_1-auc:0.945188
[110]	validation_0-auc:0.979356	validation_1-auc:0.94524
[111]	validation_0-auc:0.979588	validation_1-auc:0.945281
[112]	validation_0-auc:0.979786	validation_1-auc:0.945271
[113]	validation_0-auc:0.979883	validation_1-auc:0.945117
[114]	validation_0-auc:0.980077	validation_1-auc:0.945052
[115]	validation_0-auc:0.980207	validation_1-auc:0.945032
[116]	validation_0-auc:0.980392	validation_1-auc:0.944967
[117]	validation_0-auc:0.980604	validation_1-auc:0.945013
[118]	validation_0-auc:0.980907	validation_1-auc:0.945073
[119]	validation_0-auc:0.981193	validation_1-auc:0.945179
[120]	validation_0-auc:0.981327	validation_1-auc:0.945179
[121]	validation_0-auc:0.981364	validation_1-auc:0.945237
[122]	validation_0-auc:0.981411	validation_1-auc:0.945155
[123]	validation_0-auc:0.981443	validation_1-auc:0.945216
[124]	validation_0-auc:0.981608	validation_1-auc:0.945286
[125]	validation_0-auc:0.981736	validation_1-auc:0.945199
[126]	validation_0-auc:0.981797	validation_1-auc:0.945182
[127]	validation_0-auc:0.981911	validation_1-auc:0.945252
[128]	validation_0-auc:0.981995	validation_1-auc:0.945205
[129]	validation_0-auc:0.982135	validation_1-auc:0.945161
[130]	validation_0-auc:0.982311	validation_1-auc:0.945165
[131]	validation_0-auc:0.982455	validation_1-auc:0.945126
[132]	validation_0-auc:0.982629	validation_1-auc:0.945001
[133]	validation_0-auc:0.982874	validation_1-auc:0.944892
[134]	validation_0-auc:0.982937	validation_1-auc:0.944936
[135]	validation_0-auc:0.983076	validation_1-auc:0.944969
[136]	validation_0-auc:0.983188	validation_1-auc:0.944879
[137]	validation_0-auc:0.983287	validation_1-auc:0.944886
[138]	validation_0-auc:0.983426	validation_1-auc:0.944824
[139]	validation_0-auc:0.983525	validation_1-auc:0.944646
[140]	validation_0-auc:0.983655	validation_1-auc:0.94455
[141]	validation_0-auc:0.983714	validation_1-auc:0.944553
[142]	validation_0-auc:0.983892	validation_1-auc:0.944533
[143]	validation_0-auc:0.984032	validation_1-auc:0.944466
[144]	validation_0-auc:0.984268	validation_1-auc:0.944288
[145]	validation_0-auc:0.984389	validation_1-auc:0.944263
[146]	validation_0-auc:0.984538	validation_1-auc:0.9442
[147]	validation_0-auc:0.984685	validation_1-auc:0.944216
[148]	validation_0-auc:0.984726	validation_1-auc:0.944199
[149]	validation_0-auc:0.98486	validation_1-auc:0.944214
[150]	validation_0-auc:0.985035	validation_1-auc:0.944163
[151]	validation_0-auc:0.985211	validation_1-auc:0.944242
[152]	validation_0-auc:0.98524	validation_1-auc:0.944177
[153]	validation_0-auc:0.985381	validation_1-auc:0.944152
[154]	validation_0-auc:0.985459	validation_1-auc:0.944089
[155]	validation_0-auc:0.985607	validation_1-auc:0.94398
[156]	validation_0-auc:0.98569	validation_1-auc:0.94401
[157]	validation_0-auc:0.985724	validation_1-auc:0.944009
[158]	validation_0-auc:0.985849	validation_1-auc:0.94398
[159]	validation_0-auc:0.985963	validation_1-auc:0.943863
[160]	validation_0-auc:0.986139	validation_1-auc:0.94376
[161]	validation_0-auc:0.986226	validation_1-auc:0.94384
[162]	validation_0-auc:0.986309	validation_1-auc:0.94381
[163]	validation_0-auc:0.986415	validation_1-auc:0.943822
[164]	validation_0-auc:0.986492	validation_1-auc:0.9437
[165]	validation_0-auc:0.986731	validation_1-auc:0.943596
[166]	validation_0-auc:0.986772	validation_1-auc:0.943667
[167]	validation_0-auc:0.986866	validation_1-auc:0.943661
[168]	validation_0-auc:0.986954	validation_1-auc:0.943731
[169]	validation_0-auc:0.987075	validation_1-auc:0.943776
[170]	validation_0-auc:0.987189	validation_1-auc:0.943682
[171]	validation_0-auc:0.987291	validation_1-auc:0.94368
[172]	validation_0-auc:0.987358	validation_1-auc:0.943585
[173]	validation_0-auc:0.987501	validation_1-auc:0.943439
[174]	validation_0-auc:0.987572	validation_1-auc:0.943364
[175]	validation_0-auc:0.987651	validation_1-auc:0.943342
[176]	validation_0-auc:0.987759	validation_1-auc:0.943382
[177]	validation_0-auc:0.987872	validation_1-auc:0.943392
[178]	validation_0-auc:0.987873	validation_1-auc:0.943418
[179]	validation_0-auc:0.987975	validation_1-auc:0.943484
[180]	validation_0-auc:0.988039	validation_1-auc:0.943372
[181]	validation_0-auc:0.988076	validation_1-auc:0.943363
[182]	validation_0-auc:0.98823	validation_1-auc:0.943394
[183]	validation_0-auc:0.98836	validation_1-auc:0.943467
[184]	validation_0-auc:0.988395	validation_1-auc:0.943493
[185]	validation_0-auc:0.988573	validation_1-auc:0.94352
[186]	validation_0-auc:0.988659	validation_1-auc:0.943544
[187]	validation_0-auc:0.988727	validation_1-auc:0.943558
[188]	validation_0-auc:0.988795	validation_1-auc:0.943547
[189]	validation_0-auc:0.988954	validation_1-auc:0.943619
[190]	validation_0-auc:0.989067	validation_1-auc:0.943672
[191]	validation_0-auc:0.989197	validation_1-auc:0.943622
[192]	validation_0-auc:0.989302	validation_1-auc:0.943533
[193]	validation_0-auc:0.989448	validation_1-auc:0.943385
[194]	validation_0-auc:0.989573	validation_1-auc:0.943407
[195]	validation_0-auc:0.989628	validation_1-auc:0.943345
[196]	validation_0-auc:0.989655	validation_1-auc:0.94336
[197]	validation_0-auc:0.989687	validation_1-auc:0.943388
[198]	validation_0-auc:0.989796	validation_1-auc:0.943346
[199]	validation_0-auc:0.989907	validation_1-auc:0.943141
[200]	validation_0-auc:0.989984	validation_1-auc:0.943164
[201]	validation_0-auc:0.990116	validation_1-auc:0.943198
[202]	validation_0-auc:0.990221	validation_1-auc:0.943181
[203]	validation_0-auc:0.990291	validation_1-auc:0.943166
[204]	validation_0-auc:0.990336	validation_1-auc:0.943228
[205]	validation_0-auc:0.990477	validation_1-auc:0.943176
[206]	validation_0-auc:0.990597	validation_1-auc:0.943129
[207]	validation_0-auc:0.990719	validation_1-auc:0.943072
[208]	validation_0-auc:0.990829	validation_1-auc:0.943097
[209]	validation_0-auc:0.990923	validation_1-auc:0.94314
[210]	validation_0-auc:0.991021	validation_1-auc:0.943277
[211]	validation_0-auc:0.991118	validation_1-auc:0.943294
[212]	validation_0-auc:0.991244	validation_1-auc:0.943244
[213]	validation_0-auc:0.991309	validation_1-auc:0.943237
[214]	validation_0-auc:0.99142	validation_1-auc:0.943294
[215]	validation_0-auc:0.991551	validation_1-auc:0.943419
[216]	validation_0-auc:0.991612	validation_1-auc:0.943378
[217]	validation_0-auc:0.991682	validation_1-auc:0.943376
[218]	validation_0-auc:0.991722	validation_1-auc:0.943375
[219]	validation_0-auc:0.991768	validation_1-auc:0.943399
[220]	validation_0-auc:0.991839	validation_1-auc:0.943494
[221]	validation_0-auc:0.991917	validation_1-auc:0.94348
[222]	validation_0-auc:0.991959	validation_1-auc:0.943495
[223]	validation_0-auc:0.991971	validation_1-auc:0.943419
[224]	validation_0-auc:0.992103	validation_1-auc:0.943534
[225]	validation_0-auc:0.992145	validation_1-auc:0.94363
[226]	validation_0-auc:0.992184	validation_1-auc:0.943582
[227]	validation_0-auc:0.992204	validation_1-auc:0.943615
[228]	validation_0-auc:0.992283	validation_1-auc:0.943624
[229]	validation_0-auc:0.992358	validation_1-auc:0.943483
[230]	validation_0-auc:0.992409	validation_1-auc:0.943509
[231]	validation_0-auc:0.992449	validation_1-auc:0.943486
[232]	validation_0-auc:0.992486	validation_1-auc:0.943485
[233]	validation_0-auc:0.992516	validation_1-auc:0.943516
[234]	validation_0-auc:0.992585	validation_1-auc:0.943484
[235]	validation_0-auc:0.992732	validation_1-auc:0.943488
[236]	validation_0-auc:0.99283	validation_1-auc:0.943493
[237]	validation_0-auc:0.992882	validation_1-auc:0.943522
[238]	validation_0-auc:0.992967	validation_1-auc:0.943518
[239]	validation_0-auc:0.992986	validation_1-auc:0.943556
[240]	validation_0-auc:0.993043	validation_1-auc:0.94357
[241]	validation_0-auc:0.993069	validation_1-auc:0.94354
[242]	validation_0-auc:0.993115	validation_1-auc:0.943448
[243]	validation_0-auc:0.993176	validation_1-auc:0.943415
[244]	validation_0-auc:0.993235	validation_1-auc:0.943393
[245]	validation_0-auc:0.993312	validation_1-auc:0.943384
[246]	validation_0-auc:0.993385	validation_1-auc:0.94345
[247]	validation_0-auc:0.993416	validation_1-auc:0.943386
[248]	validation_0-auc:0.993501	validation_1-auc:0.943277
[249]	validation_0-auc:0.993539	validation_1-auc:0.943296
[250]	validation_0-auc:0.993568	validation_1-auc:0.943285
[251]	validation_0-auc:0.993649	validation_1-auc:0.943205
[252]	validation_0-auc:0.993698	validation_1-auc:0.943218
[253]	validation_0-auc:0.993757	validation_1-auc:0.943228
[254]	validation_0-auc:0.993794	validation_1-auc:0.943258
[255]	validation_0-auc:0.993848	validation_1-auc:0.943277
[256]	validation_0-auc:0.99386	validation_1-auc:0.943328
[257]	validation_0-auc:0.993937	validation_1-auc:0.943291
[258]	validation_0-auc:0.993977	validation_1-auc:0.943295
[259]	validation_0-auc:0.994044	validation_1-auc:0.943258
[260]	validation_0-auc:0.994067	validation_1-auc:0.943236
[261]	validation_0-auc:0.994068	validation_1-auc:0.943219
[262]	validation_0-auc:0.9941	validation_1-auc:0.943126
[263]	validation_0-auc:0.994157	validation_1-auc:0.943111
[264]	validation_0-auc:0.99424	validation_1-auc:0.942952
[265]	validation_0-auc:0.99426	validation_1-auc:0.942921
[266]	validation_0-auc:0.994263	validation_1-auc:0.942881
[267]	validation_0-auc:0.994314	validation_1-auc:0.942842
[268]	validation_0-auc:0.994361	validation_1-auc:0.942778
[269]	validation_0-auc:0.99444	validation_1-auc:0.942778
[270]	validation_0-auc:0.994488	validation_1-auc:0.942783
[271]	validation_0-auc:0.994545	validation_1-auc:0.942795
[272]	validation_0-auc:0.994605	validation_1-auc:0.942826
[273]	validation_0-auc:0.994633	validation_1-auc:0.942782
[274]	validation_0-auc:0.994661	validation_1-auc:0.942825
[275]	validation_0-auc:0.994707	validation_1-auc:0.942863
[276]	validation_0-auc:0.994753	validation_1-auc:0.94286
[277]	validation_0-auc:0.994788	validation_1-auc:0.942802
[278]	validation_0-auc:0.994863	validation_1-auc:0.942857
[279]	validation_0-auc:0.994901	validation_1-auc:0.942979
[280]	validation_0-auc:0.9949	validation_1-auc:0.943015
[281]	validation_0-auc:0.994938	validation_1-auc:0.942991
[282]	validation_0-auc:0.995015	validation_1-auc:0.942909
[283]	validation_0-auc:0.995052	validation_1-auc:0.942896
[284]	validation_0-auc:0.995071	validation_1-auc:0.94295
[285]	validation_0-auc:0.995123	validation_1-auc:0.942985
[286]	validation_0-auc:0.995148	validation_1-auc:0.942949
[287]	validation_0-auc:0.995136	validation_1-auc:0.943001
[288]	validation_0-auc:0.995183	validation_1-auc:0.943025
[289]	validation_0-auc:0.995211	validation_1-auc:0.943013
[290]	validation_0-auc:0.995257	validation_1-auc:0.943019
[291]	validation_0-auc:0.995307	validation_1-auc:0.942906
[292]	validation_0-auc:0.995353	validation_1-auc:0.942979
[293]	validation_0-auc:0.995426	validation_1-auc:0.942945
[294]	validation_0-auc:0.995445	validation_1-auc:0.942896
[295]	validation_0-auc:0.995482	validation_1-auc:0.94293
[296]	validation_0-auc:0.995515	validation_1-auc:0.942917
[297]	validation_0-auc:0.995525	validation_1-auc:0.942979
[298]	validation_0-auc:0.995581	validation_1-auc:0.942972
[299]	validation_0-auc:0.995615	validation_1-auc:0.943002
[300]	validation_0-auc:0.995677	validation_1-auc:0.943001
[301]	validation_0-auc:0.995697	validation_1-auc:0.942965
[302]	validation_0-auc:0.9957	validation_1-auc:0.942998
[303]	validation_0-auc:0.995712	validation_1-auc:0.942924
[304]	validation_0-auc:0.995757	validation_1-auc:0.942907
[305]	validation_0-auc:0.995804	validation_1-auc:0.942882
[306]	validation_0-auc:0.995816	validation_1-auc:0.942936
[307]	validation_0-auc:0.995847	validation_1-auc:0.942924
[308]	validation_0-auc:0.995872	validation_1-auc:0.942942
[309]	validation_0-auc:0.995894	validation_1-auc:0.942889
[310]	validation_0-auc:0.995952	validation_1-auc:0.942897
[311]	validation_0-auc:0.99596	validation_1-auc:0.942884
[312]	validation_0-auc:0.995993	validation_1-auc:0.942856
[313]	validation_0-auc:0.996079	validation_1-auc:0.942934
[314]	validation_0-auc:0.996092	validation_1-auc:0.94297
[315]	validation_0-auc:0.996131	validation_1-auc:0.942895
[316]	validation_0-auc:0.996164	validation_1-auc:0.942894
[317]	validation_0-auc:0.996197	validation_1-auc:0.9429
[318]	validation_0-auc:0.996227	validation_1-auc:0.942922
[319]	validation_0-auc:0.996247	validation_1-auc:0.942862
[320]	validation_0-auc:0.996268	validation_1-auc:0.942818
[321]	validation_0-auc:0.996289	validation_1-auc:0.942706
[322]	validation_0-auc:0.996315	validation_1-auc:0.942671
[323]	validation_0-auc:0.996354	validation_1-auc:0.942703
[324]	validation_0-auc:0.996377	validation_1-auc:0.942637
[325]	validation_0-auc:0.996404	validation_1-auc:0.942614
[326]	validation_0-auc:0.996442	validation_1-auc:0.942596
[327]	validation_0-auc:0.996457	validation_1-auc:0.942729
[328]	validation_0-auc:0.996487	validation_1-auc:0.94277
[329]	validation_0-auc:0.996518	validation_1-auc:0.942758
[330]	validation_0-auc:0.996559	validation_1-auc:0.942693
[331]	validation_0-auc:0.996613	validation_1-auc:0.942568
[332]	validation_0-auc:0.996656	validation_1-auc:0.942556
[333]	validation_0-auc:0.996681	validation_1-auc:0.942557
[334]	validation_0-auc:0.996725	validation_1-auc:0.94258
[335]	validation_0-auc:0.996763	validation_1-auc:0.942499
[336]	validation_0-auc:0.996774	validation_1-auc:0.94254
[337]	validation_0-auc:0.996819	validation_1-auc:0.942496
[338]	validation_0-auc:0.99681	validation_1-auc:0.942516
[339]	validation_0-auc:0.996815	validation_1-auc:0.942538
[340]	validation_0-auc:0.996847	validation_1-auc:0.942604
[341]	validation_0-auc:0.996877	validation_1-auc:0.942621
[342]	validation_0-auc:0.99691	validation_1-auc:0.942655
[343]	validation_0-auc:0.99696	validation_1-auc:0.942618
[344]	validation_0-auc:0.99696	validation_1-auc:0.94269
[345]	validation_0-auc:0.996985	validation_1-auc:0.942717
[346]	validation_0-auc:0.997037	validation_1-auc:0.942696
[347]	validation_0-auc:0.997058	validation_1-auc:0.942592
[348]	validation_0-auc:0.997077	validation_1-auc:0.942626
[349]	validation_0-auc:0.997115	validation_1-auc:0.942571
[350]	validation_0-auc:0.997112	validation_1-auc:0.942688
[351]	validation_0-auc:0.997129	validation_1-auc:0.942702
[352]	validation_0-auc:0.997147	validation_1-auc:0.94267
[353]	validation_0-auc:0.997163	validation_1-auc:0.942667
[354]	validation_0-auc:0.997174	validation_1-auc:0.942756
[355]	validation_0-auc:0.997183	validation_1-auc:0.942795
[356]	validation_0-auc:0.997227	validation_1-auc:0.942725
[357]	validation_0-auc:0.997238	validation_1-auc:0.942609
[358]	validation_0-auc:0.997267	validation_1-auc:0.942537
[359]	validation_0-auc:0.997285	validation_1-auc:0.942599
[360]	validation_0-auc:0.997305	validation_1-auc:0.942606
[361]	validation_0-auc:0.997316	validation_1-auc:0.942547
[362]	validation_0-auc:0.997342	validation_1-auc:0.942549
[363]	validation_0-auc:0.997389	validation_1-auc:0.942524
[364]	validation_0-auc:0.997397	validation_1-auc:0.942552
[365]	validation_0-auc:0.997419	validation_1-auc:0.94251
[366]	validation_0-auc:0.997438	validation_1-auc:0.942593
[367]	validation_0-auc:0.997488	validation_1-auc:0.942648
[368]	validation_0-auc:0.997508	validation_1-auc:0.942578
[369]	validation_0-auc:0.997503	validation_1-auc:0.94264
[370]	validation_0-auc:0.997513	validation_1-auc:0.94261
[371]	validation_0-auc:0.997521	validation_1-auc:0.94264
[372]	validation_0-auc:0.997568	validation_1-auc:0.94262
[373]	validation_0-auc:0.99759	validation_1-auc:0.942604
[374]	validation_0-auc:0.997613	validation_1-auc:0.942628
[375]	validation_0-auc:0.997611	validation_1-auc:0.942635
[376]	validation_0-auc:0.997628	validation_1-auc:0.942694
[377]	validation_0-auc:0.997658	validation_1-auc:0.942674
[378]	validation_0-auc:0.997686	validation_1-auc:0.942707
[379]	validation_0-auc:0.997702	validation_1-auc:0.942679
[380]	validation_0-auc:0.997727	validation_1-auc:0.94265
[381]	validation_0-auc:0.997735	validation_1-auc:0.942609
[382]	validation_0-auc:0.997773	validation_1-auc:0.942597
[383]	validation_0-auc:0.997795	validation_1-auc:0.942705
[384]	validation_0-auc:0.997814	validation_1-auc:0.94266
[385]	validation_0-auc:0.997828	validation_1-auc:0.942646
[386]	validation_0-auc:0.997861	validation_1-auc:0.942604
[387]	validation_0-auc:0.99789	validation_1-auc:0.942525
[388]	validation_0-auc:0.997914	validation_1-auc:0.942482
[389]	validation_0-auc:0.99794	validation_1-auc:0.942452
[390]	validation_0-auc:0.99795	validation_1-auc:0.942412
[391]	validation_0-auc:0.998009	validation_1-auc:0.942373
[392]	validation_0-auc:0.998051	validation_1-auc:0.942341
[393]	validation_0-auc:0.998069	validation_1-auc:0.942277
[394]	validation_0-auc:0.998097	validation_1-auc:0.942267
[395]	validation_0-auc:0.9981	validation_1-auc:0.942264
[396]	validation_0-auc:0.998106	validation_1-auc:0.94225
[397]	validation_0-auc:0.998128	validation_1-auc:0.942194
[398]	validation_0-auc:0.998133	validation_1-auc:0.942207
[399]	validation_0-auc:0.998136	validation_1-auc:0.942179
[400]	validation_0-auc:0.998139	validation_1-auc:0.942125
[401]	validation_0-auc:0.998191	validation_1-auc:0.942035
[402]	validation_0-auc:0.99821	validation_1-auc:0.941981
[403]	validation_0-auc:0.998225	validation_1-auc:0.941909
[404]	validation_0-auc:0.998255	validation_1-auc:0.941911
[405]	validation_0-auc:0.99826	validation_1-auc:0.941869
[406]	validation_0-auc:0.998264	validation_1-auc:0.941841
[407]	validation_0-auc:0.99827	validation_1-auc:0.941849
[408]	validation_0-auc:0.998298	validation_1-auc:0.94179
[409]	validation_0-auc:0.998327	validation_1-auc:0.941787
[410]	validation_0-auc:0.998344	validation_1-auc:0.941775
[411]	validation_0-auc:0.998343	validation_1-auc:0.941769
[412]	validation_0-auc:0.998343	validation_1-auc:0.941739
[413]	validation_0-auc:0.998378	validation_1-auc:0.941714
[414]	validation_0-auc:0.998384	validation_1-auc:0.941699
[415]	validation_0-auc:0.998384	validation_1-auc:0.941705
[416]	validation_0-auc:0.998386	validation_1-auc:0.941623
[417]	validation_0-auc:0.998403	validation_1-auc:0.941591
[418]	validation_0-auc:0.998422	validation_1-auc:0.941588
[419]	validation_0-auc:0.998444	validation_1-auc:0.941544
[420]	validation_0-auc:0.998449	validation_1-auc:0.941591
[421]	validation_0-auc:0.998467	validation_1-auc:0.941579
[422]	validation_0-auc:0.998477	validation_1-auc:0.941575
[423]	validation_0-auc:0.998495	validation_1-auc:0.941645
[424]	validation_0-auc:0.998505	validation_1-auc:0.941659
[425]	validation_0-auc:0.99851	validation_1-auc:0.941764
[426]	validation_0-auc:0.998527	validation_1-auc:0.941709
[427]	validation_0-auc:0.998541	validation_1-auc:0.941697
[428]	validation_0-auc:0.998557	validation_1-auc:0.941649
[429]	validation_0-auc:0.998572	validation_1-auc:0.941621
[430]	validation_0-auc:0.998577	validation_1-auc:0.941622
[431]	validation_0-auc:0.998587	validation_1-auc:0.941587
[432]	validation_0-auc:0.998602	validation_1-auc:0.941542
[433]	validation_0-auc:0.998627	validation_1-auc:0.941492
[434]	validation_0-auc:0.998635	validation_1-auc:0.941463
[435]	validation_0-auc:0.998646	validation_1-auc:0.941478
[436]	validation_0-auc:0.998647	validation_1-auc:0.941482
[437]	validation_0-auc:0.998678	validation_1-auc:0.941493
[438]	validation_0-auc:0.998693	validation_1-auc:0.941478
[439]	validation_0-auc:0.998702	validation_1-auc:0.941409
[440]	validation_0-auc:0.998707	validation_1-auc:0.941476
[441]	validation_0-auc:0.998699	validation_1-auc:0.941472
[442]	validation_0-auc:0.998704	validation_1-auc:0.941413
[443]	validation_0-auc:0.998712	validation_1-auc:0.941374
[444]	validation_0-auc:0.998736	validation_1-auc:0.941275
[445]	validation_0-auc:0.998739	validation_1-auc:0.941215
[446]	validation_0-auc:0.99876	validation_1-auc:0.941231
[447]	validation_0-auc:0.998769	validation_1-auc:0.941213
[448]	validation_0-auc:0.998774	validation_1-auc:0.941182
[449]	validation_0-auc:0.998793	validation_1-auc:0.941219
[450]	validation_0-auc:0.998802	validation_1-auc:0.941234
[451]	validation_0-auc:0.998816	validation_1-auc:0.941126
[452]	validation_0-auc:0.998828	validation_1-auc:0.941112
[453]	validation_0-auc:0.998837	validation_1-auc:0.941101
[454]	validation_0-auc:0.998832	validation_1-auc:0.941109
[455]	validation_0-auc:0.998842	validation_1-auc:0.941095
[456]	validation_0-auc:0.998846	validation_1-auc:0.941069
[457]	validation_0-auc:0.998857	validation_1-auc:0.941059
[458]	validation_0-auc:0.998864	validation_1-auc:0.941046
[459]	validation_0-auc:0.998887	validation_1-auc:0.94106
[460]	validation_0-auc:0.998891	validation_1-auc:0.941031
[461]	validation_0-auc:0.99891	validation_1-auc:0.941092
[462]	validation_0-auc:0.99892	validation_1-auc:0.941072
[463]	validation_0-auc:0.998944	validation_1-auc:0.940964
[464]	validation_0-auc:0.998953	validation_1-auc:0.940991
[465]	validation_0-auc:0.998969	validation_1-auc:0.940995
[466]	validation_0-auc:0.99899	validation_1-auc:0.940933
[467]	validation_0-auc:0.998991	validation_1-auc:0.940951
[468]	validation_0-auc:0.999	validation_1-auc:0.940948
[469]	validation_0-auc:0.999008	validation_1-auc:0.940926
[470]	validation_0-auc:0.999012	validation_1-auc:0.941022
[471]	validation_0-auc:0.999022	validation_1-auc:0.941048
[472]	validation_0-auc:0.999034	validation_1-auc:0.941014
[473]	validation_0-auc:0.999043	validation_1-auc:0.941057
[474]	validation_0-auc:0.999047	validation_1-auc:0.940995
[475]	validation_0-auc:0.999053	validation_1-auc:0.940984
[476]	validation_0-auc:0.999057	validation_1-auc:0.940922
[477]	validation_0-auc:0.999056	validation_1-auc:0.94088
[478]	validation_0-auc:0.999062	validation_1-auc:0.940951
[479]	validation_0-auc:0.999083	validation_1-auc:0.940995
[480]	validation_0-auc:0.999089	validation_1-auc:0.940949
[481]	validation_0-auc:0.999096	validation_1-auc:0.94093
[482]	validation_0-auc:0.999103	validation_1-auc:0.940969
[483]	validation_0-auc:0.999105	validation_1-auc:0.940976
[484]	validation_0-auc:0.999119	validation_1-auc:0.940966
[485]	validation_0-auc:0.999131	validation_1-auc:0.940913
[486]	validation_0-auc:0.999133	validation_1-auc:0.940921
[487]	validation_0-auc:0.999142	validation_1-auc:0.940938
[488]	validation_0-auc:0.999155	validation_1-auc:0.940813
[489]	validation_0-auc:0.999167	validation_1-auc:0.940793
[490]	validation_0-auc:0.999178	validation_1-auc:0.940711
[491]	validation_0-auc:0.999184	validation_1-auc:0.940752
[492]	validation_0-auc:0.99919	validation_1-auc:0.940796
[493]	validation_0-auc:0.999191	validation_1-auc:0.940854
[494]	validation_0-auc:0.999196	validation_1-auc:0.940846
[495]	validation_0-auc:0.99921	validation_1-auc:0.940763
[496]	validation_0-auc:0.999217	validation_1-auc:0.940761
[497]	validation_0-auc:0.999234	validation_1-auc:0.940634
[498]	validation_0-auc:0.999238	validation_1-auc:0.940632
[499]	validation_0-auc:0.999249	validation_1-auc:0.940589
[500]	validation_0-auc:0.999254	validation_1-auc:0.940591
[501]	validation_0-auc:0.99925	validation_1-auc:0.940561
[502]	validation_0-auc:0.999259	validation_1-auc:0.94056
[503]	validation_0-auc:0.999279	validation_1-auc:0.940572
[504]	validation_0-auc:0.999301	validation_1-auc:0.940566
[505]	validation_0-auc:0.999299	validation_1-auc:0.940558
[506]	validation_0-auc:0.999304	validation_1-auc:0.940499
[507]	validation_0-auc:0.999315	validation_1-auc:0.940444
[508]	validation_0-auc:0.99931	validation_1-auc:0.940437
[509]	validation_0-auc:0.999316	validation_1-auc:0.940425
[510]	validation_0-auc:0.999321	validation_1-auc:0.94038
[511]	validation_0-auc:0.999326	validation_1-auc:0.940338
[512]	validation_0-auc:0.999334	validation_1-auc:0.940365
[513]	validation_0-auc:0.999346	validation_1-auc:0.940381
[514]	validation_0-auc:0.999354	validation_1-auc:0.940403
[515]	validation_0-auc:0.999362	validation_1-auc:0.940354
[516]	validation_0-auc:0.99937	validation_1-auc:0.94035
[517]	validation_0-auc:0.999374	validation_1-auc:0.940417
[518]	validation_0-auc:0.999377	validation_1-auc:0.94044
[519]	validation_0-auc:0.999379	validation_1-auc:0.940375
[520]	validation_0-auc:0.999384	validation_1-auc:0.940398
[521]	validation_0-auc:0.999393	validation_1-auc:0.940422
[522]	validation_0-auc:0.999401	validation_1-auc:0.940439
[523]	validation_0-auc:0.999402	validation_1-auc:0.940412
[524]	validation_0-auc:0.999411	validation_1-auc:0.940366
[525]	validation_0-auc:0.999419	validation_1-auc:0.940376
[526]	validation_0-auc:0.999416	validation_1-auc:0.940389
[527]	validation_0-auc:0.99942	validation_1-auc:0.940433
[528]	validation_0-auc:0.999421	validation_1-auc:0.940427
[529]	validation_0-auc:0.999428	validation_1-auc:0.940404
[530]	validation_0-auc:0.999435	validation_1-auc:0.940435
[531]	validation_0-auc:0.999438	validation_1-auc:0.940466
[532]	validation_0-auc:0.999441	validation_1-auc:0.940531
[533]	validation_0-auc:0.999445	validation_1-auc:0.940536
[534]	validation_0-auc:0.999452	validation_1-auc:0.940461
[535]	validation_0-auc:0.999457	validation_1-auc:0.940486
[536]	validation_0-auc:0.999462	validation_1-auc:0.94048
[537]	validation_0-auc:0.999463	validation_1-auc:0.940434
[538]	validation_0-auc:0.999469	validation_1-auc:0.940418
[539]	validation_0-auc:0.999472	validation_1-auc:0.940423
[540]	validation_0-auc:0.999472	validation_1-auc:0.940382
[541]	validation_0-auc:0.999476	validation_1-auc:0.940368
[542]	validation_0-auc:0.999477	validation_1-auc:0.940393
[543]	validation_0-auc:0.999479	validation_1-auc:0.940402
[544]	validation_0-auc:0.999481	validation_1-auc:0.94042
[545]	validation_0-auc:0.999487	validation_1-auc:0.940404
[546]	validation_0-auc:0.999491	validation_1-auc:0.940316
[547]	validation_0-auc:0.999505	validation_1-auc:0.940274
[548]	validation_0-auc:0.999509	validation_1-auc:0.940188
[549]	validation_0-auc:0.999517	validation_1-auc:0.940167
[550]	validation_0-auc:0.999521	validation_1-auc:0.940061
[551]	validation_0-auc:0.999529	validation_1-auc:0.940121
[552]	validation_0-auc:0.999535	validation_1-auc:0.940203
[553]	validation_0-auc:0.999536	validation_1-auc:0.940201
[554]	validation_0-auc:0.999547	validation_1-auc:0.940247
[555]	validation_0-auc:0.999551	validation_1-auc:0.940229
[556]	validation_0-auc:0.999559	validation_1-auc:0.940227
[557]	validation_0-auc:0.99956	validation_1-auc:0.940233
[558]	validation_0-auc:0.999558	validation_1-auc:0.940325
[559]	validation_0-auc:0.999567	validation_1-auc:0.940279
[560]	validation_0-auc:0.999577	validation_1-auc:0.940261
[561]	validation_0-auc:0.999578	validation_1-auc:0.940251
[562]	validation_0-auc:0.999585	validation_1-auc:0.940181
[563]	validation_0-auc:0.999586	validation_1-auc:0.940138
[564]	validation_0-auc:0.999588	validation_1-auc:0.940087
[565]	validation_0-auc:0.999593	validation_1-auc:0.940023
[566]	validation_0-auc:0.999598	validation_1-auc:0.94
[567]	validation_0-auc:0.999604	validation_1-auc:0.940036
[568]	validation_0-auc:0.999608	validation_1-auc:0.940033
[569]	validation_0-auc:0.999613	validation_1-auc:0.94005
[570]	validation_0-auc:0.999622	validation_1-auc:0.939959
[571]	validation_0-auc:0.999625	validation_1-auc:0.939919
[572]	validation_0-auc:0.999624	validation_1-auc:0.939915
[573]	validation_0-auc:0.999628	validation_1-auc:0.93986
[574]	validation_0-auc:0.999637	validation_1-auc:0.939911
[575]	validation_0-auc:0.999642	validation_1-auc:0.939973
[576]	validation_0-auc:0.999642	validation_1-auc:0.940041
[577]	validation_0-auc:0.999645	validation_1-auc:0.939991
[578]	validation_0-auc:0.999647	validation_1-auc:0.939984
[579]	validation_0-auc:0.999652	validation_1-auc:0.939989
[580]	validation_0-auc:0.999652	validation_1-auc:0.939983
[581]	validation_0-auc:0.999658	validation_1-auc:0.939993
[582]	validation_0-auc:0.999664	validation_1-auc:0.940018
[583]	validation_0-auc:0.999669	validation_1-auc:0.940005
[584]	validation_0-auc:0.999673	validation_1-auc:0.940007
[585]	validation_0-auc:0.999675	validation_1-auc:0.939924
[586]	validation_0-auc:0.999683	validation_1-auc:0.939891
[587]	validation_0-auc:0.999682	validation_1-auc:0.9399
[588]	validation_0-auc:0.99968	validation_1-auc:0.939901
[589]	validation_0-auc:0.999683	validation_1-auc:0.939922
[590]	validation_0-auc:0.999682	validation_1-auc:0.939915
[591]	validation_0-auc:0.999692	validation_1-auc:0.939922
[592]	validation_0-auc:0.999693	validation_1-auc:0.939859
[593]	validation_0-auc:0.999698	validation_1-auc:0.939805
[594]	validation_0-auc:0.999704	validation_1-auc:0.939771
[595]	validation_0-auc:0.999712	validation_1-auc:0.939797
[596]	validation_0-auc:0.999715	validation_1-auc:0.939701
[597]	validation_0-auc:0.999717	validation_1-auc:0.939652
[598]	validation_0-auc:0.999717	validation_1-auc:0.939631
[599]	validation_0-auc:0.999718	validation_1-auc:0.939632
[600]	validation_0-auc:0.999722	validation_1-auc:0.939663
[601]	validation_0-auc:0.999725	validation_1-auc:0.939639
[602]	validation_0-auc:0.999731	validation_1-auc:0.939661
[603]	validation_0-auc:0.999728	validation_1-auc:0.939683
[604]	validation_0-auc:0.999731	validation_1-auc:0.93967
[605]	validation_0-auc:0.999736	validation_1-auc:0.939711
[606]	validation_0-auc:0.999737	validation_1-auc:0.93968
[607]	validation_0-auc:0.99974	validation_1-auc:0.939606
[608]	validation_0-auc:0.999744	validation_1-auc:0.939647
[609]	validation_0-auc:0.999746	validation_1-auc:0.9396
[610]	validation_0-auc:0.999749	validation_1-auc:0.939591
[611]	validation_0-auc:0.999755	validation_1-auc:0.939629
[612]	validation_0-auc:0.999757	validation_1-auc:0.939676
[613]	validation_0-auc:0.999757	validation_1-auc:0.939706
[614]	validation_0-auc:0.99976	validation_1-auc:0.939669
[615]	validation_0-auc:0.999762	validation_1-auc:0.939633
[616]	validation_0-auc:0.999764	validation_1-auc:0.939633
[617]	validation_0-auc:0.999766	validation_1-auc:0.939631
[618]	validation_0-auc:0.999769	validation_1-auc:0.93963
[619]	validation_0-auc:0.999769	validation_1-auc:0.939565
[620]	validation_0-auc:0.99977	validation_1-auc:0.939574
[621]	validation_0-auc:0.999773	validation_1-auc:0.939534
[622]	validation_0-auc:0.999771	validation_1-auc:0.939526
[623]	validation_0-auc:0.999775	validation_1-auc:0.939492
[624]	validation_0-auc:0.999776	validation_1-auc:0.939499
[625]	validation_0-auc:0.999776	validation_1-auc:0.939535
[626]	validation_0-auc:0.999777	validation_1-auc:0.939481
[627]	validation_0-auc:0.999781	validation_1-auc:0.939431
[628]	validation_0-auc:0.999781	validation_1-auc:0.939425
[629]	validation_0-auc:0.999783	validation_1-auc:0.939416
[630]	validation_0-auc:0.999782	validation_1-auc:0.939369
[631]	validation_0-auc:0.999785	validation_1-auc:0.939348
[632]	validation_0-auc:0.999784	validation_1-auc:0.939403
[633]	validation_0-auc:0.999784	validation_1-auc:0.939375
[634]	validation_0-auc:0.999788	validation_1-auc:0.93936
[635]	validation_0-auc:0.99979	validation_1-auc:0.939363
[636]	validation_0-auc:0.999793	validation_1-auc:0.939449
[637]	validation_0-auc:0.999798	validation_1-auc:0.939427
[638]	validation_0-auc:0.999802	validation_1-auc:0.939446
[639]	validation_0-auc:0.999803	validation_1-auc:0.939452
[640]	validation_0-auc:0.999807	validation_1-auc:0.939454
[641]	validation_0-auc:0.999807	validation_1-auc:0.939413
[642]	validation_0-auc:0.999806	validation_1-auc:0.939431
[643]	validation_0-auc:0.999807	validation_1-auc:0.939422
[644]	validation_0-auc:0.999806	validation_1-auc:0.939508
[645]	validation_0-auc:0.999811	validation_1-auc:0.939513
[646]	validation_0-auc:0.999814	validation_1-auc:0.939527
[647]	validation_0-auc:0.999814	validation_1-auc:0.939508
[648]	validation_0-auc:0.999817	validation_1-auc:0.939499
[649]	validation_0-auc:0.99982	validation_1-auc:0.939519
[650]	validation_0-auc:0.999824	validation_1-auc:0.939523
[651]	validation_0-auc:0.999825	validation_1-auc:0.93956
[652]	validation_0-auc:0.999827	validation_1-auc:0.93954
[653]	validation_0-auc:0.999828	validation_1-auc:0.939535
[654]	validation_0-auc:0.999827	validation_1-auc:0.939454
[655]	validation_0-auc:0.999829	validation_1-auc:0.939419
[656]	validation_0-auc:0.99983	validation_1-auc:0.939395
[657]	validation_0-auc:0.999831	validation_1-auc:0.939445
[658]	validation_0-auc:0.999831	validation_1-auc:0.939407
[659]	validation_0-auc:0.99983	validation_1-auc:0.939407
[660]	validation_0-auc:0.999834	validation_1-auc:0.939359
[661]	validation_0-auc:0.999836	validation_1-auc:0.939392
[662]	validation_0-auc:0.999838	validation_1-auc:0.93937
[663]	validation_0-auc:0.999838	validation_1-auc:0.939376
[664]	validation_0-auc:0.999838	validation_1-auc:0.939388
[665]	validation_0-auc:0.99984	validation_1-auc:0.939402
[666]	validation_0-auc:0.999841	validation_1-auc:0.939358
[667]	validation_0-auc:0.99984	validation_1-auc:0.939374
[668]	validation_0-auc:0.999841	validation_1-auc:0.939362
[669]	validation_0-auc:0.999842	validation_1-auc:0.939328
[670]	validation_0-auc:0.999841	validation_1-auc:0.939376
[671]	validation_0-auc:0.999842	validation_1-auc:0.939362
[672]	validation_0-auc:0.999844	validation_1-auc:0.939319
[673]	validation_0-auc:0.999846	validation_1-auc:0.939288
[674]	validation_0-auc:0.999845	validation_1-auc:0.939257
[675]	validation_0-auc:0.999845	validation_1-auc:0.939275
[676]	validation_0-auc:0.999845	validation_1-auc:0.939253
[677]	validation_0-auc:0.999847	validation_1-auc:0.939178
[678]	validation_0-auc:0.999849	validation_1-auc:0.939162
[679]	validation_0-auc:0.999848	validation_1-auc:0.93917
[680]	validation_0-auc:0.99985	validation_1-auc:0.939232
[681]	validation_0-auc:0.999851	validation_1-auc:0.939256
[682]	validation_0-auc:0.99985	validation_1-auc:0.939226
[683]	validation_0-auc:0.999855	validation_1-auc:0.939203
[684]	validation_0-auc:0.999856	validation_1-auc:0.939176
[685]	validation_0-auc:0.99986	validation_1-auc:0.939168
[686]	validation_0-auc:0.999859	validation_1-auc:0.939188
[687]	validation_0-auc:0.99986	validation_1-auc:0.939234
[688]	validation_0-auc:0.999862	validation_1-auc:0.939185
[689]	validation_0-auc:0.999864	validation_1-auc:0.93916
[690]	validation_0-auc:0.999866	validation_1-auc:0.939193
[691]	validation_0-auc:0.999866	validation_1-auc:0.939177
[692]	validation_0-auc:0.999868	validation_1-auc:0.939139
[693]	validation_0-auc:0.999871	validation_1-auc:0.939133
[694]	validation_0-auc:0.999869	validation_1-auc:0.93916
[695]	validation_0-auc:0.999869	validation_1-auc:0.939153
[696]	validation_0-auc:0.999872	validation_1-auc:0.939128
[697]	validation_0-auc:0.999876	validation_1-auc:0.939214
[698]	validation_0-auc:0.999875	validation_1-auc:0.939191
[699]	validation_0-auc:0.999874	validation_1-auc:0.939211
[700]	validation_0-auc:0.999873	validation_1-auc:0.939236
[701]	validation_0-auc:0.999879	validation_1-auc:0.939287
[702]	validation_0-auc:0.99988	validation_1-auc:0.939238
[703]	validation_0-auc:0.999881	validation_1-auc:0.939211
[704]	validation_0-auc:0.999881	validation_1-auc:0.939192
[705]	validation_0-auc:0.999884	validation_1-auc:0.939188
[706]	validation_0-auc:0.999883	validation_1-auc:0.939141
[707]	validation_0-auc:0.999883	validation_1-auc:0.93913
[708]	validation_0-auc:0.999887	validation_1-auc:0.939149
[709]	validation_0-auc:0.999887	validation_1-auc:0.939203
[710]	validation_0-auc:0.999888	validation_1-auc:0.939185
[711]	validation_0-auc:0.999888	validation_1-auc:0.939174
[712]	validation_0-auc:0.999891	validation_1-auc:0.939098
[713]	validation_0-auc:0.999892	validation_1-auc:0.939085
[714]	validation_0-auc:0.999892	validation_1-auc:0.939016
[715]	validation_0-auc:0.999895	validation_1-auc:0.939003
[716]	validation_0-auc:0.999895	validation_1-auc:0.938966
[717]	validation_0-auc:0.999895	validation_1-auc:0.938899
[718]	validation_0-auc:0.999898	validation_1-auc:0.938818
[719]	validation_0-auc:0.999898	validation_1-auc:0.938851
[720]	validation_0-auc:0.999899	validation_1-auc:0.93888
[721]	validation_0-auc:0.999898	validation_1-auc:0.938844
[722]	validation_0-auc:0.999899	validation_1-auc:0.938818
[723]	validation_0-auc:0.9999	validation_1-auc:0.938839
[724]	validation_0-auc:0.9999	validation_1-auc:0.938819
[725]	validation_0-auc:0.999899	validation_1-auc:0.938793
[726]	validation_0-auc:0.9999	validation_1-auc:0.938708
[727]	validation_0-auc:0.999901	validation_1-auc:0.938691
[728]	validation_0-auc:0.999902	validation_1-auc:0.938726
[729]	validation_0-auc:0.999903	validation_1-auc:0.938776
[730]	validation_0-auc:0.999904	validation_1-auc:0.938826
[731]	validation_0-auc:0.999905	validation_1-auc:0.938833
[732]	validation_0-auc:0.99991	validation_1-auc:0.938853
[733]	validation_0-auc:0.999909	validation_1-auc:0.938854
[734]	validation_0-auc:0.999909	validation_1-auc:0.938837
[735]	validation_0-auc:0.999911	validation_1-auc:0.938771
[736]	validation_0-auc:0.999909	validation_1-auc:0.938803
[737]	validation_0-auc:0.99991	validation_1-auc:0.93877
[738]	validation_0-auc:0.99991	validation_1-auc:0.938723
[739]	validation_0-auc:0.999911	validation_1-auc:0.93866
[740]	validation_0-auc:0.999911	validation_1-auc:0.938625
[741]	validation_0-auc:0.999914	validation_1-auc:0.938609
[742]	validation_0-auc:0.999915	validation_1-auc:0.938614
[743]	validation_0-auc:0.999915	validation_1-auc:0.938556
[744]	validation_0-auc:0.999918	validation_1-auc:0.938493
[745]	validation_0-auc:0.99992	validation_1-auc:0.938482
[746]	validation_0-auc:0.999919	validation_1-auc:0.938482
[747]	validation_0-auc:0.99992	validation_1-auc:0.938468
[748]	validation_0-auc:0.999922	validation_1-auc:0.938448
[749]	validation_0-auc:0.999925	validation_1-auc:0.938447
[750]	validation_0-auc:0.999924	validation_1-auc:0.938435
[751]	validation_0-auc:0.999924	validation_1-auc:0.93842
[752]	validation_0-auc:0.999924	validation_1-auc:0.938385
[753]	validation_0-auc:0.999924	validation_1-auc:0.938362
[754]	validation_0-auc:0.999924	validation_1-auc:0.938406
[755]	validation_0-auc:0.999925	validation_1-auc:0.938364
[756]	validation_0-auc:0.999925	validation_1-auc:0.938366
[757]	validation_0-auc:0.999924	validation_1-auc:0.938383
[758]	validation_0-auc:0.999924	validation_1-auc:0.938358
[759]	validation_0-auc:0.999925	validation_1-auc:0.938361
[760]	validation_0-auc:0.999923	validation_1-auc:0.938358
[761]	validation_0-auc:0.999925	validation_1-auc:0.938375
[762]	validation_0-auc:0.999925	validation_1-auc:0.938362
[763]	validation_0-auc:0.999925	validation_1-auc:0.93837
[764]	validation_0-auc:0.999927	validation_1-auc:0.938419
[765]	validation_0-auc:0.999928	validation_1-auc:0.938346
[766]	validation_0-auc:0.999927	validation_1-auc:0.938294
[767]	validation_0-auc:0.999926	validation_1-auc:0.938256
[768]	validation_0-auc:0.999927	validation_1-auc:0.938253
[769]	validation_0-auc:0.999929	validation_1-auc:0.938268
[770]	validation_0-auc:0.999931	validation_1-auc:0.938249
[771]	validation_0-auc:0.999932	validation_1-auc:0.938269
[772]	validation_0-auc:0.999934	validation_1-auc:0.938274
[773]	validation_0-auc:0.999935	validation_1-auc:0.938252
[774]	validation_0-auc:0.999936	validation_1-auc:0.938299
[775]	validation_0-auc:0.999937	validation_1-auc:0.938274
[776]	validation_0-auc:0.999937	validation_1-auc:0.938236
[777]	validation_0-auc:0.999938	validation_1-auc:0.938299
[778]	validation_0-auc:0.999939	validation_1-auc:0.938307
[779]	validation_0-auc:0.99994	validation_1-auc:0.93836
[780]	validation_0-auc:0.99994	validation_1-auc:0.938359
[781]	validation_0-auc:0.999939	validation_1-auc:0.938384
[782]	validation_0-auc:0.999941	validation_1-auc:0.93843
[783]	validation_0-auc:0.999941	validation_1-auc:0.938415
[784]	validation_0-auc:0.999944	validation_1-auc:0.938389
[785]	validation_0-auc:0.999946	validation_1-auc:0.93839
[786]	validation_0-auc:0.999946	validation_1-auc:0.938357
[787]	validation_0-auc:0.999946	validation_1-auc:0.9383
[788]	validation_0-auc:0.999946	validation_1-auc:0.93828
[789]	validation_0-auc:0.999947	validation_1-auc:0.938244
[790]	validation_0-auc:0.999948	validation_1-auc:0.938246
[791]	validation_0-auc:0.999953	validation_1-auc:0.9382
[792]	validation_0-auc:0.999954	validation_1-auc:0.938177
[793]	validation_0-auc:0.999954	validation_1-auc:0.938164
[794]	validation_0-auc:0.999953	validation_1-auc:0.93809
[795]	validation_0-auc:0.999954	validation_1-auc:0.938106
[796]	validation_0-auc:0.999954	validation_1-auc:0.938125
[797]	validation_0-auc:0.999953	validation_1-auc:0.938141
[798]	validation_0-auc:0.999954	validation_1-auc:0.938173
[799]	validation_0-auc:0.999956	validation_1-auc:0.938212
[800]	validation_0-auc:0.999956	validation_1-auc:0.938207
[801]	validation_0-auc:0.999955	validation_1-auc:0.938299
[802]	validation_0-auc:0.999955	validation_1-auc:0.938283
[803]	validation_0-auc:0.999955	validation_1-auc:0.938246
[804]	validation_0-auc:0.999956	validation_1-auc:0.938243
[805]	validation_0-auc:0.999956	validation_1-auc:0.938193
[806]	validation_0-auc:0.999957	validation_1-auc:0.938194
[807]	validation_0-auc:0.999958	validation_1-auc:0.938168
[808]	validation_0-auc:0.999958	validation_1-auc:0.938166
[809]	validation_0-auc:0.999959	validation_1-auc:0.93816
[810]	validation_0-auc:0.999958	validation_1-auc:0.938174
[811]	validation_0-auc:0.99996	validation_1-auc:0.938152
[812]	validation_0-auc:0.99996	validation_1-auc:0.93812
[813]	validation_0-auc:0.999959	validation_1-auc:0.938103
[814]	validation_0-auc:0.999959	validation_1-auc:0.938079
[815]	validation_0-auc:0.999959	validation_1-auc:0.938117
[816]	validation_0-auc:0.999961	validation_1-auc:0.938145
[817]	validation_0-auc:0.999963	validation_1-auc:0.938138
[818]	validation_0-auc:0.999964	validation_1-auc:0.938157
[819]	validation_0-auc:0.999964	validation_1-auc:0.938212
[820]	validation_0-auc:0.999965	validation_1-auc:0.938192
[821]	validation_0-auc:0.999965	validation_1-auc:0.938156
[822]	validation_0-auc:0.999966	validation_1-auc:0.938147
[823]	validation_0-auc:0.999965	validation_1-auc:0.938149
[824]	validation_0-auc:0.999965	validation_1-auc:0.938098
[825]	validation_0-auc:0.999967	validation_1-auc:0.938138
[826]	validation_0-auc:0.999968	validation_1-auc:0.938095
[827]	validation_0-auc:0.999969	validation_1-auc:0.938104
[828]	validation_0-auc:0.999968	validation_1-auc:0.938065
[829]	validation_0-auc:0.999967	validation_1-auc:0.93804
[830]	validation_0-auc:0.999967	validation_1-auc:0.93809
[831]	validation_0-auc:0.999968	validation_1-auc:0.938098
[832]	validation_0-auc:0.999968	validation_1-auc:0.938074
[833]	validation_0-auc:0.999969	validation_1-auc:0.938098
[834]	validation_0-auc:0.99997	validation_1-auc:0.93809
[835]	validation_0-auc:0.999971	validation_1-auc:0.938094
[836]	validation_0-auc:0.999971	validation_1-auc:0.938094
[837]	validation_0-auc:0.999971	validation_1-auc:0.938094
[838]	validation_0-auc:0.999973	validation_1-auc:0.93811
[839]	validation_0-auc:0.999972	validation_1-auc:0.93814
[840]	validation_0-auc:0.999972	validation_1-auc:0.938183
[841]	validation_0-auc:0.999972	validation_1-auc:0.938183
[842]	validation_0-auc:0.999974	validation_1-auc:0.93813
[843]	validation_0-auc:0.999974	validation_1-auc:0.938134
[844]	validation_0-auc:0.999973	validation_1-auc:0.938139
[845]	validation_0-auc:0.999972	validation_1-auc:0.938182
[846]	validation_0-auc:0.999973	validation_1-auc:0.938053
[847]	validation_0-auc:0.999974	validation_1-auc:0.937965
[848]	validation_0-auc:0.999975	validation_1-auc:0.937927
[849]	validation_0-auc:0.999975	validation_1-auc:0.937878
[850]	validation_0-auc:0.999975	validation_1-auc:0.937887
[851]	validation_0-auc:0.999976	validation_1-auc:0.937928
[852]	validation_0-auc:0.999978	validation_1-auc:0.9379
[853]	validation_0-auc:0.999978	validation_1-auc:0.937932
[854]	validation_0-auc:0.999978	validation_1-auc:0.937929
[855]	validation_0-auc:0.999978	validation_1-auc:0.937974
[856]	validation_0-auc:0.999978	validation_1-auc:0.938004
[857]	validation_0-auc:0.999978	validation_1-auc:0.938007
[858]	validation_0-auc:0.99998	validation_1-auc:0.938018
[859]	validation_0-auc:0.99998	validation_1-auc:0.938006
[860]	validation_0-auc:0.99998	validation_1-auc:0.93797
[861]	validation_0-auc:0.99998	validation_1-auc:0.938
[862]	validation_0-auc:0.999981	validation_1-auc:0.938002
[863]	validation_0-auc:0.999981	validation_1-auc:0.938042
[864]	validation_0-auc:0.999981	validation_1-auc:0.938053
[865]	validation_0-auc:0.999981	validation_1-auc:0.937986
[866]	validation_0-auc:0.999982	validation_1-auc:0.937976
[867]	validation_0-auc:0.999982	validation_1-auc:0.937945
[868]	validation_0-auc:0.999982	validation_1-auc:0.937953
[869]	validation_0-auc:0.999982	validation_1-auc:0.937993
[870]	validation_0-auc:0.999982	validation_1-auc:0.937979
[871]	validation_0-auc:0.999983	validation_1-auc:0.937994
[872]	validation_0-auc:0.999982	validation_1-auc:0.938029
[873]	validation_0-auc:0.999982	validation_1-auc:0.938035
[874]	validation_0-auc:0.999983	validation_1-auc:0.937968
[875]	validation_0-auc:0.999983	validation_1-auc:0.937994
[876]	validation_0-auc:0.999983	validation_1-auc:0.937953
[877]	validation_0-auc:0.999983	validation_1-auc:0.937929
[878]	validation_0-auc:0.999983	validation_1-auc:0.937953
[879]	validation_0-auc:0.999983	validation_1-auc:0.937894
[880]	validation_0-auc:0.999983	validation_1-auc:0.937949
[881]	validation_0-auc:0.999985	validation_1-auc:0.937929
[882]	validation_0-auc:0.999984	validation_1-auc:0.937966
[883]	validation_0-auc:0.999985	validation_1-auc:0.93791
[884]	validation_0-auc:0.999986	validation_1-auc:0.937928
[885]	validation_0-auc:0.999987	validation_1-auc:0.937883
[886]	validation_0-auc:0.999988	validation_1-auc:0.937834
[887]	validation_0-auc:0.999988	validation_1-auc:0.937833
[888]	validation_0-auc:0.999988	validation_1-auc:0.937783
[889]	validation_0-auc:0.999988	validation_1-auc:0.937771
[890]	validation_0-auc:0.999989	validation_1-auc:0.937756
[891]	validation_0-auc:0.999989	validation_1-auc:0.937747
[892]	validation_0-auc:0.999989	validation_1-auc:0.937758
[893]	validation_0-auc:0.99999	validation_1-auc:0.937733
[894]	validation_0-auc:0.999989	validation_1-auc:0.937742
[895]	validation_0-auc:0.99999	validation_1-auc:0.937743
[896]	validation_0-auc:0.99999	validation_1-auc:0.937727
[897]	validation_0-auc:0.99999	validation_1-auc:0.937726
[898]	validation_0-auc:0.99999	validation_1-auc:0.937711
[899]	validation_0-auc:0.99999	validation_1-auc:0.937719
[900]	validation_0-auc:0.999991	validation_1-auc:0.937684
[901]	validation_0-auc:0.999992	validation_1-auc:0.937686
[902]	validation_0-auc:0.999991	validation_1-auc:0.937683
[903]	validation_0-auc:0.999991	validation_1-auc:0.937679
[904]	validation_0-auc:0.999991	validation_1-auc:0.93768
[905]	validation_0-auc:0.99999	validation_1-auc:0.937727
[906]	validation_0-auc:0.999992	validation_1-auc:0.937714
[907]	validation_0-auc:0.999992	validation_1-auc:0.937776
[908]	validation_0-auc:0.999991	validation_1-auc:0.937794
[909]	validation_0-auc:0.999993	validation_1-auc:0.937754
[910]	validation_0-auc:0.999993	validation_1-auc:0.937777
[911]	validation_0-auc:0.999993	validation_1-auc:0.937756
[912]	validation_0-auc:0.999994	validation_1-auc:0.937758
[913]	validation_0-auc:0.999994	validation_1-auc:0.937791
[914]	validation_0-auc:0.999993	validation_1-auc:0.937807
[915]	validation_0-auc:0.999994	validation_1-auc:0.937775
[916]	validation_0-auc:0.999994	validation_1-auc:0.937772
[917]	validation_0-auc:0.999994	validation_1-auc:0.937758
[918]	validation_0-auc:0.999995	validation_1-auc:0.937798
[919]	validation_0-auc:0.999995	validation_1-auc:0.937793
[920]	validation_0-auc:0.999994	validation_1-auc:0.937831
[921]	validation_0-auc:0.999994	validation_1-auc:0.937779
[922]	validation_0-auc:0.999993	validation_1-auc:0.937812
[923]	validation_0-auc:0.999994	validation_1-auc:0.937836
[924]	validation_0-auc:0.999994	validation_1-auc:0.937868
[925]	validation_0-auc:0.999994	validation_1-auc:0.937806
[926]	validation_0-auc:0.999994	validation_1-auc:0.937781
[927]	validation_0-auc:0.999994	validation_1-auc:0.937773
[928]	validation_0-auc:0.999994	validation_1-auc:0.937732
[929]	validation_0-auc:0.999994	validation_1-auc:0.93774
[930]	validation_0-auc:0.999994	validation_1-auc:0.937696
[931]	validation_0-auc:0.999994	validation_1-auc:0.937684
[932]	validation_0-auc:0.999994	validation_1-auc:0.937711
[933]	validation_0-auc:0.999994	validation_1-auc:0.937696
[934]	validation_0-auc:0.999994	validation_1-auc:0.937711
[935]	validation_0-auc:0.999994	validation_1-auc:0.9377
[936]	validation_0-auc:0.999995	validation_1-auc:0.937655
[937]	validation_0-auc:0.999995	validation_1-auc:0.937661
[938]	validation_0-auc:0.999995	validation_1-auc:0.937686
[939]	validation_0-auc:0.999995	validation_1-auc:0.937659
[940]	validation_0-auc:0.999995	validation_1-auc:0.937733
[941]	validation_0-auc:0.999995	validation_1-auc:0.937741
[942]	validation_0-auc:0.999995	validation_1-auc:0.937764
[943]	validation_0-auc:0.999994	validation_1-auc:0.937723
[944]	validation_0-auc:0.999994	validation_1-auc:0.937719
[945]	validation_0-auc:0.999995	validation_1-auc:0.937692
[946]	validation_0-auc:0.999995	validation_1-auc:0.937678
[947]	validation_0-auc:0.999994	validation_1-auc:0.937685
[948]	validation_0-auc:0.999995	validation_1-auc:0.937708
[949]	validation_0-auc:0.999995	validation_1-auc:0.937681
[950]	validation_0-auc:0.999995	validation_1-auc:0.937665
[951]	validation_0-auc:0.999996	validation_1-auc:0.937653
[952]	validation_0-auc:0.999995	validation_1-auc:0.93764
[953]	validation_0-auc:0.999995	validation_1-auc:0.937634
[954]	validation_0-auc:0.999995	validation_1-auc:0.937617
[955]	validation_0-auc:0.999995	validation_1-auc:0.937619
[956]	validation_0-auc:0.999995	validation_1-auc:0.937617
[957]	validation_0-auc:0.999996	validation_1-auc:0.937654
[958]	validation_0-auc:0.999996	validation_1-auc:0.937632
[959]	validation_0-auc:0.999996	validation_1-auc:0.9376
[960]	validation_0-auc:0.999996	validation_1-auc:0.937584
[961]	validation_0-auc:0.999996	validation_1-auc:0.937579
[962]	validation_0-auc:0.999996	validation_1-auc:0.937565
[963]	validation_0-auc:0.999996	validation_1-auc:0.937552
[964]	validation_0-auc:0.999996	validation_1-auc:0.937612
[965]	validation_0-auc:0.999996	validation_1-auc:0.937675
[966]	validation_0-auc:0.999996	validation_1-auc:0.937695
[967]	validation_0-auc:0.999996	validation_1-auc:0.9377
[968]	validation_0-auc:0.999996	validation_1-auc:0.937675
[969]	validation_0-auc:0.999996	validation_1-auc:0.937645
[970]	validation_0-auc:0.999996	validation_1-auc:0.937618
[971]	validation_0-auc:0.999996	validation_1-auc:0.937592
[972]	validation_0-auc:0.999997	validation_1-auc:0.937624
[973]	validation_0-auc:0.999997	validation_1-auc:0.937633
[974]	validation_0-auc:0.999997	validation_1-auc:0.937681
[975]	validation_0-auc:0.999997	validation_1-auc:0.937619
[976]	validation_0-auc:0.999997	validation_1-auc:0.937631
[977]	validation_0-auc:0.999997	validation_1-auc:0.937591
[978]	validation_0-auc:0.999997	validation_1-auc:0.93763
[979]	validation_0-auc:0.999997	validation_1-auc:0.937635
[980]	validation_0-auc:0.999997	validation_1-auc:0.937614
[981]	validation_0-auc:0.999998	validation_1-auc:0.937618
[982]	validation_0-auc:0.999998	validation_1-auc:0.937598
[983]	validation_0-auc:0.999998	validation_1-auc:0.937595
[984]	validation_0-auc:0.999998	validation_1-auc:0.937592
[985]	validation_0-auc:0.999998	validation_1-auc:0.937618
[986]	validation_0-auc:0.999997	validation_1-auc:0.937604
[987]	validation_0-auc:0.999998	validation_1-auc:0.937624
[988]	validation_0-auc:0.999998	validation_1-auc:0.937585
[989]	validation_0-auc:0.999998	validation_1-auc:0.937571
[990]	validation_0-auc:0.999998	validation_1-auc:0.937566
[991]	validation_0-auc:0.999998	validation_1-auc:0.937547
[992]	validation_0-auc:0.999998	validation_1-auc:0.937529
[993]	validation_0-auc:0.999998	validation_1-auc:0.937534
[994]	validation_0-auc:0.999998	validation_1-auc:0.937472
[995]	validation_0-auc:0.999998	validation_1-auc:0.937445
[996]	validation_0-auc:0.999998	validation_1-auc:0.937438
[997]	validation_0-auc:0.999998	validation_1-auc:0.937442
[998]	validation_0-auc:0.999998	validation_1-auc:0.937561
[999]	validation_0-auc:0.999998	validation_1-auc:0.937556
Wall time: 33.9 s
Out[92]:
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1,
              disable_default_eval_metric=1, eval_metric='auc', gamma=0.15,
              learning_rate=0.1, max_delta_step=0, max_depth=7,
              min_child_weight=3, missing=None, n_estimators=1000, n_jobs=8,
              nthread=None, objective='binary:logistic', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=7.8243170862346005,
              seed=None, silent=None, subsample=0.5, verbose_eval=100,
              verbosity=0)
In [93]:
true_flag = test_df_total['y']
pred_flag = clf1.predict(test_df_total[features])

print("AUC score: {:<8.3f}".format(roc_auc_score(true_flag, pred_flag)))
AUC score: 0.806   

Check confusion matrix, AUC plot and other metrics

In [94]:
metrics = confusion_mat_plot(true_flag, pred_flag)
accuracy: 90.37 %
precision: 55.05 %
Recall: 67.99 %
specificity: 93.14 %
F-score: 60.84 %
In [95]:
gini = auc_plot(test_df_total[features], true_flag, clf1)

# record metrics
model_comparison = pd.DataFrame()

model_comparison['With Eng Feat'] = [gini, metrics[0], metrics[1], metrics[2], metrics[3], metrics[4]]
model_comparison.index = ['Gini', 'Accuracy', 'Precision', 'Recall', 'Specificity', 'F-Score']
gini 87.511 %

Let's check the feature importance.

In [96]:
feature_importance_df = pd.DataFrame()
feature_importance_df["Feature"] = features
feature_importance_df["importance"] = clf1.feature_importances_

# plot results
plt.figure(figsize=(12,12))
sns.barplot(x="importance", y="Feature", data=feature_importance_df.sort_values(by="importance",ascending=False))
plt.title('Features importance')
plt.tight_layout()
plt.show()

It seems that most of the features that we engineered are the most important ones to decide the split.

So, as for the next steps, we will build three models, with the same hyperparameters for training and compare the performance:

  • One with the top 5 features from the previous model

  • We will try to make our model more explainable, and check if we sacrifice in performance. This means that we will try the same model for dataset that does not contain the engineered features and check performance

  • Next we will also choose the top 5 features from that model too

Model 2 - Top 5 features

In [97]:
%%time
clf2 = xgb.XGBClassifier(**best_parameters)
top_5_feat = feature_importance_df.sort_values(by="importance",ascending=False)['Feature'].values[:5]
tr_val = (train_df_total[top_5_feat], target)
ts_val = (test_df_total[top_5_feat], test_df_total['y'])

clf2.fit(train_df_total[top_5_feat], target, eval_metric='auc', eval_set=[tr_val, ts_val])
[0]	validation_0-auc:0.932246	validation_1-auc:0.926305
[1]	validation_0-auc:0.936538	validation_1-auc:0.930017
[2]	validation_0-auc:0.936971	validation_1-auc:0.929717
[3]	validation_0-auc:0.938608	validation_1-auc:0.932285
[4]	validation_0-auc:0.942368	validation_1-auc:0.937251
[5]	validation_0-auc:0.94405	validation_1-auc:0.938275
[6]	validation_0-auc:0.945183	validation_1-auc:0.938988
[7]	validation_0-auc:0.945584	validation_1-auc:0.939349
[8]	validation_0-auc:0.945868	validation_1-auc:0.939238
[9]	validation_0-auc:0.946375	validation_1-auc:0.939687
[10]	validation_0-auc:0.946521	validation_1-auc:0.939771
[11]	validation_0-auc:0.946923	validation_1-auc:0.939901
[12]	validation_0-auc:0.947251	validation_1-auc:0.940129
[13]	validation_0-auc:0.947483	validation_1-auc:0.94021
[14]	validation_0-auc:0.947632	validation_1-auc:0.939982
[15]	validation_0-auc:0.947797	validation_1-auc:0.940019
[16]	validation_0-auc:0.947983	validation_1-auc:0.940331
[17]	validation_0-auc:0.948189	validation_1-auc:0.940281
[18]	validation_0-auc:0.948383	validation_1-auc:0.940308
[19]	validation_0-auc:0.948541	validation_1-auc:0.940538
[20]	validation_0-auc:0.948811	validation_1-auc:0.94073
[21]	validation_0-auc:0.948913	validation_1-auc:0.940628
[22]	validation_0-auc:0.949147	validation_1-auc:0.940678
[23]	validation_0-auc:0.949359	validation_1-auc:0.94058
[24]	validation_0-auc:0.949464	validation_1-auc:0.940731
[25]	validation_0-auc:0.949639	validation_1-auc:0.94085
[26]	validation_0-auc:0.94988	validation_1-auc:0.940908
[27]	validation_0-auc:0.949949	validation_1-auc:0.940702
[28]	validation_0-auc:0.950032	validation_1-auc:0.940616
[29]	validation_0-auc:0.950096	validation_1-auc:0.940666
[30]	validation_0-auc:0.950205	validation_1-auc:0.940714
[31]	validation_0-auc:0.95035	validation_1-auc:0.940738
[32]	validation_0-auc:0.950515	validation_1-auc:0.940875
[33]	validation_0-auc:0.950624	validation_1-auc:0.9409
[34]	validation_0-auc:0.951109	validation_1-auc:0.941249
[35]	validation_0-auc:0.951206	validation_1-auc:0.941281
[36]	validation_0-auc:0.9514	validation_1-auc:0.94119
[37]	validation_0-auc:0.95158	validation_1-auc:0.940974
[38]	validation_0-auc:0.951681	validation_1-auc:0.941036
[39]	validation_0-auc:0.951798	validation_1-auc:0.941022
[40]	validation_0-auc:0.951892	validation_1-auc:0.940962
[41]	validation_0-auc:0.952039	validation_1-auc:0.941085
[42]	validation_0-auc:0.952132	validation_1-auc:0.941001
[43]	validation_0-auc:0.952144	validation_1-auc:0.941055
[44]	validation_0-auc:0.952232	validation_1-auc:0.940936
[45]	validation_0-auc:0.952349	validation_1-auc:0.940845
[46]	validation_0-auc:0.952566	validation_1-auc:0.940708
[47]	validation_0-auc:0.952636	validation_1-auc:0.940661
[48]	validation_0-auc:0.952697	validation_1-auc:0.940655
[49]	validation_0-auc:0.952856	validation_1-auc:0.94071
[50]	validation_0-auc:0.953022	validation_1-auc:0.940616
[51]	validation_0-auc:0.953081	validation_1-auc:0.940578
[52]	validation_0-auc:0.953236	validation_1-auc:0.940583
[53]	validation_0-auc:0.953324	validation_1-auc:0.940615
[54]	validation_0-auc:0.953416	validation_1-auc:0.940743
[55]	validation_0-auc:0.953498	validation_1-auc:0.940728
[56]	validation_0-auc:0.953586	validation_1-auc:0.940652
[57]	validation_0-auc:0.953746	validation_1-auc:0.940638
[58]	validation_0-auc:0.953926	validation_1-auc:0.940652
[59]	validation_0-auc:0.954053	validation_1-auc:0.940667
[60]	validation_0-auc:0.954122	validation_1-auc:0.940725
[61]	validation_0-auc:0.954196	validation_1-auc:0.940697
[62]	validation_0-auc:0.954422	validation_1-auc:0.94067
[63]	validation_0-auc:0.954534	validation_1-auc:0.940574
[64]	validation_0-auc:0.954646	validation_1-auc:0.940534
[65]	validation_0-auc:0.954737	validation_1-auc:0.940459
[66]	validation_0-auc:0.954882	validation_1-auc:0.940432
[67]	validation_0-auc:0.95501	validation_1-auc:0.940433
[68]	validation_0-auc:0.955045	validation_1-auc:0.940398
[69]	validation_0-auc:0.95523	validation_1-auc:0.940455
[70]	validation_0-auc:0.955324	validation_1-auc:0.940472
[71]	validation_0-auc:0.955468	validation_1-auc:0.940381
[72]	validation_0-auc:0.955594	validation_1-auc:0.940314
[73]	validation_0-auc:0.955729	validation_1-auc:0.940219
[74]	validation_0-auc:0.955799	validation_1-auc:0.940166
[75]	validation_0-auc:0.955899	validation_1-auc:0.940086
[76]	validation_0-auc:0.95599	validation_1-auc:0.940123
[77]	validation_0-auc:0.956095	validation_1-auc:0.940146
[78]	validation_0-auc:0.956193	validation_1-auc:0.940142
[79]	validation_0-auc:0.956361	validation_1-auc:0.940118
[80]	validation_0-auc:0.956488	validation_1-auc:0.940145
[81]	validation_0-auc:0.956542	validation_1-auc:0.940169
[82]	validation_0-auc:0.956626	validation_1-auc:0.940148
[83]	validation_0-auc:0.956831	validation_1-auc:0.940071
[84]	validation_0-auc:0.956916	validation_1-auc:0.939945
[85]	validation_0-auc:0.957091	validation_1-auc:0.939779
[86]	validation_0-auc:0.957211	validation_1-auc:0.939638
[87]	validation_0-auc:0.957224	validation_1-auc:0.93964
[88]	validation_0-auc:0.957239	validation_1-auc:0.93961
[89]	validation_0-auc:0.957382	validation_1-auc:0.939699
[90]	validation_0-auc:0.957507	validation_1-auc:0.939662
[91]	validation_0-auc:0.957598	validation_1-auc:0.939667
[92]	validation_0-auc:0.95763	validation_1-auc:0.939684
[93]	validation_0-auc:0.957659	validation_1-auc:0.93974
[94]	validation_0-auc:0.957764	validation_1-auc:0.939745
[95]	validation_0-auc:0.957888	validation_1-auc:0.939702
[96]	validation_0-auc:0.957945	validation_1-auc:0.93969
[97]	validation_0-auc:0.95804	validation_1-auc:0.939662
[98]	validation_0-auc:0.958097	validation_1-auc:0.939668
[99]	validation_0-auc:0.958226	validation_1-auc:0.939557
[100]	validation_0-auc:0.958279	validation_1-auc:0.939631
[101]	validation_0-auc:0.958452	validation_1-auc:0.93946
[102]	validation_0-auc:0.958505	validation_1-auc:0.939457
[103]	validation_0-auc:0.958534	validation_1-auc:0.939474
[104]	validation_0-auc:0.958692	validation_1-auc:0.939386
[105]	validation_0-auc:0.95887	validation_1-auc:0.939292
[106]	validation_0-auc:0.95891	validation_1-auc:0.939233
[107]	validation_0-auc:0.958966	validation_1-auc:0.939135
[108]	validation_0-auc:0.959056	validation_1-auc:0.939026
[109]	validation_0-auc:0.959077	validation_1-auc:0.939039
[110]	validation_0-auc:0.959118	validation_1-auc:0.939017
[111]	validation_0-auc:0.959165	validation_1-auc:0.939004
[112]	validation_0-auc:0.959193	validation_1-auc:0.939039
[113]	validation_0-auc:0.959272	validation_1-auc:0.939102
[114]	validation_0-auc:0.959437	validation_1-auc:0.939079
[115]	validation_0-auc:0.959608	validation_1-auc:0.939101
[116]	validation_0-auc:0.959719	validation_1-auc:0.939128
[117]	validation_0-auc:0.959719	validation_1-auc:0.939066
[118]	validation_0-auc:0.959918	validation_1-auc:0.939145
[119]	validation_0-auc:0.960008	validation_1-auc:0.93911
[120]	validation_0-auc:0.960062	validation_1-auc:0.939024
[121]	validation_0-auc:0.960159	validation_1-auc:0.939037
[122]	validation_0-auc:0.960275	validation_1-auc:0.939183
[123]	validation_0-auc:0.960365	validation_1-auc:0.939102
[124]	validation_0-auc:0.960481	validation_1-auc:0.938989
[125]	validation_0-auc:0.960532	validation_1-auc:0.939046
[126]	validation_0-auc:0.960637	validation_1-auc:0.938924
[127]	validation_0-auc:0.960661	validation_1-auc:0.938945
[128]	validation_0-auc:0.960721	validation_1-auc:0.938865
[129]	validation_0-auc:0.960785	validation_1-auc:0.938782
[130]	validation_0-auc:0.960946	validation_1-auc:0.938735
[131]	validation_0-auc:0.961068	validation_1-auc:0.938685
[132]	validation_0-auc:0.96123	validation_1-auc:0.938717
[133]	validation_0-auc:0.96127	validation_1-auc:0.938666
[134]	validation_0-auc:0.961269	validation_1-auc:0.938638
[135]	validation_0-auc:0.961359	validation_1-auc:0.938585
[136]	validation_0-auc:0.961488	validation_1-auc:0.93853
[137]	validation_0-auc:0.961554	validation_1-auc:0.938449
[138]	validation_0-auc:0.961566	validation_1-auc:0.938478
[139]	validation_0-auc:0.961683	validation_1-auc:0.938366
[140]	validation_0-auc:0.961801	validation_1-auc:0.938316
[141]	validation_0-auc:0.961895	validation_1-auc:0.938389
[142]	validation_0-auc:0.962012	validation_1-auc:0.93843
[143]	validation_0-auc:0.962064	validation_1-auc:0.938373
[144]	validation_0-auc:0.962171	validation_1-auc:0.93841
[145]	validation_0-auc:0.962226	validation_1-auc:0.938328
[146]	validation_0-auc:0.962283	validation_1-auc:0.938369
[147]	validation_0-auc:0.962347	validation_1-auc:0.938411
[148]	validation_0-auc:0.962446	validation_1-auc:0.938351
[149]	validation_0-auc:0.962455	validation_1-auc:0.938356
[150]	validation_0-auc:0.962574	validation_1-auc:0.93838
[151]	validation_0-auc:0.962635	validation_1-auc:0.93837
[152]	validation_0-auc:0.962684	validation_1-auc:0.938433
[153]	validation_0-auc:0.962816	validation_1-auc:0.938325
[154]	validation_0-auc:0.962887	validation_1-auc:0.9384
[155]	validation_0-auc:0.962979	validation_1-auc:0.938411
[156]	validation_0-auc:0.96301	validation_1-auc:0.938368
[157]	validation_0-auc:0.963087	validation_1-auc:0.938329
[158]	validation_0-auc:0.963227	validation_1-auc:0.938216
[159]	validation_0-auc:0.963288	validation_1-auc:0.938309
[160]	validation_0-auc:0.963407	validation_1-auc:0.938275
[161]	validation_0-auc:0.963456	validation_1-auc:0.93824
[162]	validation_0-auc:0.963486	validation_1-auc:0.938267
[163]	validation_0-auc:0.963516	validation_1-auc:0.938281
[164]	validation_0-auc:0.963661	validation_1-auc:0.93804
[165]	validation_0-auc:0.963761	validation_1-auc:0.937989
[166]	validation_0-auc:0.963796	validation_1-auc:0.938029
[167]	validation_0-auc:0.96389	validation_1-auc:0.937923
[168]	validation_0-auc:0.963978	validation_1-auc:0.937834
[169]	validation_0-auc:0.964126	validation_1-auc:0.937871
[170]	validation_0-auc:0.964223	validation_1-auc:0.937818
[171]	validation_0-auc:0.964293	validation_1-auc:0.937749
[172]	validation_0-auc:0.96435	validation_1-auc:0.937712
[173]	validation_0-auc:0.964358	validation_1-auc:0.937676
[174]	validation_0-auc:0.964349	validation_1-auc:0.937657
[175]	validation_0-auc:0.964466	validation_1-auc:0.937735
[176]	validation_0-auc:0.964528	validation_1-auc:0.937602
[177]	validation_0-auc:0.96457	validation_1-auc:0.937694
[178]	validation_0-auc:0.964664	validation_1-auc:0.937665
[179]	validation_0-auc:0.964705	validation_1-auc:0.937674
[180]	validation_0-auc:0.964757	validation_1-auc:0.937748
[181]	validation_0-auc:0.964882	validation_1-auc:0.937742
[182]	validation_0-auc:0.964992	validation_1-auc:0.937651
[183]	validation_0-auc:0.964999	validation_1-auc:0.937595
[184]	validation_0-auc:0.965097	validation_1-auc:0.937576
[185]	validation_0-auc:0.965198	validation_1-auc:0.937623
[186]	validation_0-auc:0.965294	validation_1-auc:0.93751
[187]	validation_0-auc:0.965389	validation_1-auc:0.937509
[188]	validation_0-auc:0.965439	validation_1-auc:0.937536
[189]	validation_0-auc:0.965534	validation_1-auc:0.937673
[190]	validation_0-auc:0.965576	validation_1-auc:0.937621
[191]	validation_0-auc:0.965675	validation_1-auc:0.937594
[192]	validation_0-auc:0.965673	validation_1-auc:0.937552
[193]	validation_0-auc:0.965786	validation_1-auc:0.937522
[194]	validation_0-auc:0.965848	validation_1-auc:0.937497
[195]	validation_0-auc:0.965894	validation_1-auc:0.937469
[196]	validation_0-auc:0.965926	validation_1-auc:0.937414
[197]	validation_0-auc:0.965989	validation_1-auc:0.937401
[198]	validation_0-auc:0.966051	validation_1-auc:0.937445
[199]	validation_0-auc:0.966079	validation_1-auc:0.93754
[200]	validation_0-auc:0.966185	validation_1-auc:0.937401
[201]	validation_0-auc:0.966279	validation_1-auc:0.937393
[202]	validation_0-auc:0.966401	validation_1-auc:0.937374
[203]	validation_0-auc:0.96652	validation_1-auc:0.93724
[204]	validation_0-auc:0.966642	validation_1-auc:0.937078
[205]	validation_0-auc:0.966801	validation_1-auc:0.936931
[206]	validation_0-auc:0.966833	validation_1-auc:0.936909
[207]	validation_0-auc:0.966863	validation_1-auc:0.93686
[208]	validation_0-auc:0.966944	validation_1-auc:0.936811
[209]	validation_0-auc:0.967042	validation_1-auc:0.936848
[210]	validation_0-auc:0.9671	validation_1-auc:0.93693
[211]	validation_0-auc:0.967109	validation_1-auc:0.936876
[212]	validation_0-auc:0.967194	validation_1-auc:0.936819
[213]	validation_0-auc:0.967224	validation_1-auc:0.936821
[214]	validation_0-auc:0.967289	validation_1-auc:0.936792
[215]	validation_0-auc:0.967356	validation_1-auc:0.936694
[216]	validation_0-auc:0.967379	validation_1-auc:0.936651
[217]	validation_0-auc:0.967432	validation_1-auc:0.936684
[218]	validation_0-auc:0.967466	validation_1-auc:0.936566
[219]	validation_0-auc:0.967491	validation_1-auc:0.936511
[220]	validation_0-auc:0.967609	validation_1-auc:0.936474
[221]	validation_0-auc:0.967685	validation_1-auc:0.936382
[222]	validation_0-auc:0.967749	validation_1-auc:0.936402
[223]	validation_0-auc:0.967849	validation_1-auc:0.936398
[224]	validation_0-auc:0.967867	validation_1-auc:0.936337
[225]	validation_0-auc:0.967942	validation_1-auc:0.936298
[226]	validation_0-auc:0.968	validation_1-auc:0.936346
[227]	validation_0-auc:0.968032	validation_1-auc:0.936343
[228]	validation_0-auc:0.968093	validation_1-auc:0.936383
[229]	validation_0-auc:0.968139	validation_1-auc:0.936448
[230]	validation_0-auc:0.968209	validation_1-auc:0.936359
[231]	validation_0-auc:0.968277	validation_1-auc:0.936308
[232]	validation_0-auc:0.968309	validation_1-auc:0.936304
[233]	validation_0-auc:0.968405	validation_1-auc:0.93622
[234]	validation_0-auc:0.968518	validation_1-auc:0.936163
[235]	validation_0-auc:0.968611	validation_1-auc:0.93608
[236]	validation_0-auc:0.968659	validation_1-auc:0.936071
[237]	validation_0-auc:0.968681	validation_1-auc:0.936059
[238]	validation_0-auc:0.968733	validation_1-auc:0.93605
[239]	validation_0-auc:0.968809	validation_1-auc:0.936042
[240]	validation_0-auc:0.968861	validation_1-auc:0.936057
[241]	validation_0-auc:0.968944	validation_1-auc:0.936046
[242]	validation_0-auc:0.969032	validation_1-auc:0.936057
[243]	validation_0-auc:0.969051	validation_1-auc:0.936032
[244]	validation_0-auc:0.969054	validation_1-auc:0.936035
[245]	validation_0-auc:0.96907	validation_1-auc:0.936024
[246]	validation_0-auc:0.96919	validation_1-auc:0.935951
[247]	validation_0-auc:0.969323	validation_1-auc:0.935886
[248]	validation_0-auc:0.969407	validation_1-auc:0.935787
[249]	validation_0-auc:0.969422	validation_1-auc:0.935751
[250]	validation_0-auc:0.969438	validation_1-auc:0.935745
[251]	validation_0-auc:0.969496	validation_1-auc:0.935756
[252]	validation_0-auc:0.969521	validation_1-auc:0.935803
[253]	validation_0-auc:0.969622	validation_1-auc:0.935804
[254]	validation_0-auc:0.969661	validation_1-auc:0.935795
[255]	validation_0-auc:0.969691	validation_1-auc:0.935844
[256]	validation_0-auc:0.969793	validation_1-auc:0.935691
[257]	validation_0-auc:0.969866	validation_1-auc:0.935826
[258]	validation_0-auc:0.969926	validation_1-auc:0.93576
[259]	validation_0-auc:0.969929	validation_1-auc:0.935714
[260]	validation_0-auc:0.970022	validation_1-auc:0.935663
[261]	validation_0-auc:0.970105	validation_1-auc:0.935477
[262]	validation_0-auc:0.970214	validation_1-auc:0.935391
[263]	validation_0-auc:0.970249	validation_1-auc:0.935417
[264]	validation_0-auc:0.970273	validation_1-auc:0.93544
[265]	validation_0-auc:0.970303	validation_1-auc:0.935358
[266]	validation_0-auc:0.970405	validation_1-auc:0.935365
[267]	validation_0-auc:0.970476	validation_1-auc:0.935341
[268]	validation_0-auc:0.970514	validation_1-auc:0.93529
[269]	validation_0-auc:0.970616	validation_1-auc:0.935267
[270]	validation_0-auc:0.970688	validation_1-auc:0.935233
[271]	validation_0-auc:0.970782	validation_1-auc:0.935166
[272]	validation_0-auc:0.970836	validation_1-auc:0.935067
[273]	validation_0-auc:0.970913	validation_1-auc:0.935092
[274]	validation_0-auc:0.970976	validation_1-auc:0.935015
[275]	validation_0-auc:0.971032	validation_1-auc:0.935052
[276]	validation_0-auc:0.971046	validation_1-auc:0.935056
[277]	validation_0-auc:0.971083	validation_1-auc:0.935015
[278]	validation_0-auc:0.971184	validation_1-auc:0.934882
[279]	validation_0-auc:0.971193	validation_1-auc:0.934901
[280]	validation_0-auc:0.971219	validation_1-auc:0.934936
[281]	validation_0-auc:0.97126	validation_1-auc:0.934875
[282]	validation_0-auc:0.971363	validation_1-auc:0.934811
[283]	validation_0-auc:0.971411	validation_1-auc:0.934789
[284]	validation_0-auc:0.971463	validation_1-auc:0.934739
[285]	validation_0-auc:0.9715	validation_1-auc:0.934741
[286]	validation_0-auc:0.971525	validation_1-auc:0.934652
[287]	validation_0-auc:0.971576	validation_1-auc:0.93474
[288]	validation_0-auc:0.971642	validation_1-auc:0.934686
[289]	validation_0-auc:0.971681	validation_1-auc:0.934649
[290]	validation_0-auc:0.971765	validation_1-auc:0.934643
[291]	validation_0-auc:0.971872	validation_1-auc:0.934613
[292]	validation_0-auc:0.971893	validation_1-auc:0.934719
[293]	validation_0-auc:0.971954	validation_1-auc:0.934668
[294]	validation_0-auc:0.972018	validation_1-auc:0.934658
[295]	validation_0-auc:0.972042	validation_1-auc:0.934575
[296]	validation_0-auc:0.972136	validation_1-auc:0.934617
[297]	validation_0-auc:0.972176	validation_1-auc:0.934596
[298]	validation_0-auc:0.97221	validation_1-auc:0.934581
[299]	validation_0-auc:0.972238	validation_1-auc:0.934556
[300]	validation_0-auc:0.972325	validation_1-auc:0.934516
[301]	validation_0-auc:0.972317	validation_1-auc:0.934512
[302]	validation_0-auc:0.972368	validation_1-auc:0.934438
[303]	validation_0-auc:0.972441	validation_1-auc:0.934309
[304]	validation_0-auc:0.972524	validation_1-auc:0.934342
[305]	validation_0-auc:0.972545	validation_1-auc:0.9344
[306]	validation_0-auc:0.972626	validation_1-auc:0.934391
[307]	validation_0-auc:0.972686	validation_1-auc:0.934333
[308]	validation_0-auc:0.972698	validation_1-auc:0.934442
[309]	validation_0-auc:0.972719	validation_1-auc:0.934422
[310]	validation_0-auc:0.972774	validation_1-auc:0.934362
[311]	validation_0-auc:0.972808	validation_1-auc:0.934408
[312]	validation_0-auc:0.972807	validation_1-auc:0.934473
[313]	validation_0-auc:0.972912	validation_1-auc:0.934324
[314]	validation_0-auc:0.972969	validation_1-auc:0.934302
[315]	validation_0-auc:0.973038	validation_1-auc:0.934248
[316]	validation_0-auc:0.973088	validation_1-auc:0.934189
[317]	validation_0-auc:0.973136	validation_1-auc:0.934261
[318]	validation_0-auc:0.973159	validation_1-auc:0.934229
[319]	validation_0-auc:0.973265	validation_1-auc:0.93409
[320]	validation_0-auc:0.973346	validation_1-auc:0.934102
[321]	validation_0-auc:0.973411	validation_1-auc:0.934029
[322]	validation_0-auc:0.97342	validation_1-auc:0.934154
[323]	validation_0-auc:0.973426	validation_1-auc:0.934193
[324]	validation_0-auc:0.973436	validation_1-auc:0.934186
[325]	validation_0-auc:0.973471	validation_1-auc:0.934139
[326]	validation_0-auc:0.973504	validation_1-auc:0.934075
[327]	validation_0-auc:0.973545	validation_1-auc:0.934114
[328]	validation_0-auc:0.973617	validation_1-auc:0.934059
[329]	validation_0-auc:0.973655	validation_1-auc:0.934097
[330]	validation_0-auc:0.973659	validation_1-auc:0.934144
[331]	validation_0-auc:0.973654	validation_1-auc:0.934112
[332]	validation_0-auc:0.973719	validation_1-auc:0.934103
[333]	validation_0-auc:0.973767	validation_1-auc:0.93414
[334]	validation_0-auc:0.973858	validation_1-auc:0.934158
[335]	validation_0-auc:0.973955	validation_1-auc:0.933965
[336]	validation_0-auc:0.973994	validation_1-auc:0.933951
[337]	validation_0-auc:0.974078	validation_1-auc:0.933849
[338]	validation_0-auc:0.974099	validation_1-auc:0.933854
[339]	validation_0-auc:0.974192	validation_1-auc:0.933764
[340]	validation_0-auc:0.97428	validation_1-auc:0.933709
[341]	validation_0-auc:0.97429	validation_1-auc:0.933725
[342]	validation_0-auc:0.97437	validation_1-auc:0.933671
[343]	validation_0-auc:0.974368	validation_1-auc:0.933682
[344]	validation_0-auc:0.974373	validation_1-auc:0.933703
[345]	validation_0-auc:0.97445	validation_1-auc:0.933746
[346]	validation_0-auc:0.974482	validation_1-auc:0.933748
[347]	validation_0-auc:0.974529	validation_1-auc:0.93368
[348]	validation_0-auc:0.974541	validation_1-auc:0.933654
[349]	validation_0-auc:0.974649	validation_1-auc:0.933671
[350]	validation_0-auc:0.974723	validation_1-auc:0.933579
[351]	validation_0-auc:0.974788	validation_1-auc:0.933523
[352]	validation_0-auc:0.974869	validation_1-auc:0.933421
[353]	validation_0-auc:0.974924	validation_1-auc:0.933474
[354]	validation_0-auc:0.974924	validation_1-auc:0.933439
[355]	validation_0-auc:0.974946	validation_1-auc:0.933451
[356]	validation_0-auc:0.975014	validation_1-auc:0.933476
[357]	validation_0-auc:0.975061	validation_1-auc:0.933444
[358]	validation_0-auc:0.975112	validation_1-auc:0.933428
[359]	validation_0-auc:0.975092	validation_1-auc:0.933396
[360]	validation_0-auc:0.975157	validation_1-auc:0.93341
[361]	validation_0-auc:0.975182	validation_1-auc:0.933509
[362]	validation_0-auc:0.975204	validation_1-auc:0.933563
[363]	validation_0-auc:0.975239	validation_1-auc:0.93341
[364]	validation_0-auc:0.975258	validation_1-auc:0.933307
[365]	validation_0-auc:0.975282	validation_1-auc:0.93329
[366]	validation_0-auc:0.975304	validation_1-auc:0.933334
[367]	validation_0-auc:0.97531	validation_1-auc:0.93336
[368]	validation_0-auc:0.975362	validation_1-auc:0.93325
[369]	validation_0-auc:0.975374	validation_1-auc:0.933281
[370]	validation_0-auc:0.975363	validation_1-auc:0.933292
[371]	validation_0-auc:0.975401	validation_1-auc:0.933199
[372]	validation_0-auc:0.975416	validation_1-auc:0.93325
[373]	validation_0-auc:0.975453	validation_1-auc:0.933223
[374]	validation_0-auc:0.975442	validation_1-auc:0.933246
[375]	validation_0-auc:0.975445	validation_1-auc:0.933327
[376]	validation_0-auc:0.975444	validation_1-auc:0.933373
[377]	validation_0-auc:0.975514	validation_1-auc:0.933409
[378]	validation_0-auc:0.975562	validation_1-auc:0.933398
[379]	validation_0-auc:0.975594	validation_1-auc:0.933391
[380]	validation_0-auc:0.975594	validation_1-auc:0.933393
[381]	validation_0-auc:0.975684	validation_1-auc:0.933398
[382]	validation_0-auc:0.975728	validation_1-auc:0.933333
[383]	validation_0-auc:0.975752	validation_1-auc:0.93335
[384]	validation_0-auc:0.975787	validation_1-auc:0.933358
[385]	validation_0-auc:0.975829	validation_1-auc:0.933345
[386]	validation_0-auc:0.975878	validation_1-auc:0.933361
[387]	validation_0-auc:0.975901	validation_1-auc:0.933364
[388]	validation_0-auc:0.97599	validation_1-auc:0.933307
[389]	validation_0-auc:0.975994	validation_1-auc:0.933383
[390]	validation_0-auc:0.976004	validation_1-auc:0.933453
[391]	validation_0-auc:0.976035	validation_1-auc:0.933535
[392]	validation_0-auc:0.976071	validation_1-auc:0.933631
[393]	validation_0-auc:0.976102	validation_1-auc:0.93365
[394]	validation_0-auc:0.976174	validation_1-auc:0.933674
[395]	validation_0-auc:0.976234	validation_1-auc:0.933737
[396]	validation_0-auc:0.976265	validation_1-auc:0.933793
[397]	validation_0-auc:0.976312	validation_1-auc:0.933802
[398]	validation_0-auc:0.976345	validation_1-auc:0.933851
[399]	validation_0-auc:0.976395	validation_1-auc:0.933809
[400]	validation_0-auc:0.976419	validation_1-auc:0.933754
[401]	validation_0-auc:0.976457	validation_1-auc:0.933769
[402]	validation_0-auc:0.976521	validation_1-auc:0.933661
[403]	validation_0-auc:0.976557	validation_1-auc:0.933673
[404]	validation_0-auc:0.976623	validation_1-auc:0.933618
[405]	validation_0-auc:0.976715	validation_1-auc:0.933507
[406]	validation_0-auc:0.976732	validation_1-auc:0.933557
[407]	validation_0-auc:0.976779	validation_1-auc:0.933604
[408]	validation_0-auc:0.976868	validation_1-auc:0.933491
[409]	validation_0-auc:0.976938	validation_1-auc:0.933418
[410]	validation_0-auc:0.977005	validation_1-auc:0.93338
[411]	validation_0-auc:0.977053	validation_1-auc:0.933366
[412]	validation_0-auc:0.977071	validation_1-auc:0.933318
[413]	validation_0-auc:0.977125	validation_1-auc:0.933299
[414]	validation_0-auc:0.977121	validation_1-auc:0.933312
[415]	validation_0-auc:0.97712	validation_1-auc:0.933322
[416]	validation_0-auc:0.977117	validation_1-auc:0.933258
[417]	validation_0-auc:0.977117	validation_1-auc:0.933369
[418]	validation_0-auc:0.977181	validation_1-auc:0.933182
[419]	validation_0-auc:0.977198	validation_1-auc:0.933145
[420]	validation_0-auc:0.977254	validation_1-auc:0.933162
[421]	validation_0-auc:0.977291	validation_1-auc:0.933209
[422]	validation_0-auc:0.977344	validation_1-auc:0.933198
[423]	validation_0-auc:0.977385	validation_1-auc:0.933252
[424]	validation_0-auc:0.97745	validation_1-auc:0.933203
[425]	validation_0-auc:0.977427	validation_1-auc:0.933206
[426]	validation_0-auc:0.977453	validation_1-auc:0.933144
[427]	validation_0-auc:0.97749	validation_1-auc:0.933137
[428]	validation_0-auc:0.977505	validation_1-auc:0.933199
[429]	validation_0-auc:0.977521	validation_1-auc:0.933131
[430]	validation_0-auc:0.977554	validation_1-auc:0.933059
[431]	validation_0-auc:0.977555	validation_1-auc:0.933125
[432]	validation_0-auc:0.977579	validation_1-auc:0.933177
[433]	validation_0-auc:0.977586	validation_1-auc:0.933244
[434]	validation_0-auc:0.977667	validation_1-auc:0.933102
[435]	validation_0-auc:0.977702	validation_1-auc:0.933158
[436]	validation_0-auc:0.97772	validation_1-auc:0.93318
[437]	validation_0-auc:0.977748	validation_1-auc:0.933169
[438]	validation_0-auc:0.977754	validation_1-auc:0.933125
[439]	validation_0-auc:0.977776	validation_1-auc:0.933159
[440]	validation_0-auc:0.977851	validation_1-auc:0.933112
[441]	validation_0-auc:0.977838	validation_1-auc:0.933027
[442]	validation_0-auc:0.97789	validation_1-auc:0.933016
[443]	validation_0-auc:0.97793	validation_1-auc:0.932985
[444]	validation_0-auc:0.977934	validation_1-auc:0.932981
[445]	validation_0-auc:0.978007	validation_1-auc:0.932907
[446]	validation_0-auc:0.978034	validation_1-auc:0.932876
[447]	validation_0-auc:0.978069	validation_1-auc:0.932886
[448]	validation_0-auc:0.978088	validation_1-auc:0.932814
[449]	validation_0-auc:0.978133	validation_1-auc:0.932811
[450]	validation_0-auc:0.978207	validation_1-auc:0.932732
[451]	validation_0-auc:0.978271	validation_1-auc:0.93268
[452]	validation_0-auc:0.978347	validation_1-auc:0.932641
[453]	validation_0-auc:0.978402	validation_1-auc:0.932703
[454]	validation_0-auc:0.978405	validation_1-auc:0.932697
[455]	validation_0-auc:0.978404	validation_1-auc:0.932771
[456]	validation_0-auc:0.97844	validation_1-auc:0.932761
[457]	validation_0-auc:0.978489	validation_1-auc:0.932744
[458]	validation_0-auc:0.978497	validation_1-auc:0.932694
[459]	validation_0-auc:0.978516	validation_1-auc:0.932706
[460]	validation_0-auc:0.978543	validation_1-auc:0.932729
[461]	validation_0-auc:0.978595	validation_1-auc:0.932725
[462]	validation_0-auc:0.978618	validation_1-auc:0.932709
[463]	validation_0-auc:0.978645	validation_1-auc:0.932712
[464]	validation_0-auc:0.978678	validation_1-auc:0.932747
[465]	validation_0-auc:0.978685	validation_1-auc:0.932787
[466]	validation_0-auc:0.97874	validation_1-auc:0.932732
[467]	validation_0-auc:0.978798	validation_1-auc:0.932779
[468]	validation_0-auc:0.978799	validation_1-auc:0.932808
[469]	validation_0-auc:0.978861	validation_1-auc:0.932797
[470]	validation_0-auc:0.978838	validation_1-auc:0.932935
[471]	validation_0-auc:0.978875	validation_1-auc:0.932882
[472]	validation_0-auc:0.978907	validation_1-auc:0.932816
[473]	validation_0-auc:0.97891	validation_1-auc:0.932893
[474]	validation_0-auc:0.9789	validation_1-auc:0.932962
[475]	validation_0-auc:0.978921	validation_1-auc:0.932966
[476]	validation_0-auc:0.978905	validation_1-auc:0.933018
[477]	validation_0-auc:0.978899	validation_1-auc:0.933042
[478]	validation_0-auc:0.978941	validation_1-auc:0.933106
[479]	validation_0-auc:0.978969	validation_1-auc:0.933119
[480]	validation_0-auc:0.979036	validation_1-auc:0.932996
[481]	validation_0-auc:0.979043	validation_1-auc:0.933052
[482]	validation_0-auc:0.979076	validation_1-auc:0.93307
[483]	validation_0-auc:0.979142	validation_1-auc:0.933068
[484]	validation_0-auc:0.979233	validation_1-auc:0.933152
[485]	validation_0-auc:0.979297	validation_1-auc:0.93306
[486]	validation_0-auc:0.97936	validation_1-auc:0.93292
[487]	validation_0-auc:0.979381	validation_1-auc:0.932881
[488]	validation_0-auc:0.979421	validation_1-auc:0.932908
[489]	validation_0-auc:0.979464	validation_1-auc:0.932957
[490]	validation_0-auc:0.979478	validation_1-auc:0.932977
[491]	validation_0-auc:0.979496	validation_1-auc:0.933093
[492]	validation_0-auc:0.979521	validation_1-auc:0.93314
[493]	validation_0-auc:0.979565	validation_1-auc:0.933109
[494]	validation_0-auc:0.979637	validation_1-auc:0.93308
[495]	validation_0-auc:0.979676	validation_1-auc:0.933135
[496]	validation_0-auc:0.979646	validation_1-auc:0.933225
[497]	validation_0-auc:0.979765	validation_1-auc:0.93317
[498]	validation_0-auc:0.979843	validation_1-auc:0.933163
[499]	validation_0-auc:0.979887	validation_1-auc:0.933143
[500]	validation_0-auc:0.979922	validation_1-auc:0.933051
[501]	validation_0-auc:0.980006	validation_1-auc:0.933076
[502]	validation_0-auc:0.980022	validation_1-auc:0.933157
[503]	validation_0-auc:0.980103	validation_1-auc:0.933044
[504]	validation_0-auc:0.980115	validation_1-auc:0.933081
[505]	validation_0-auc:0.980122	validation_1-auc:0.933126
[506]	validation_0-auc:0.98016	validation_1-auc:0.933146
[507]	validation_0-auc:0.980184	validation_1-auc:0.93314
[508]	validation_0-auc:0.980167	validation_1-auc:0.933138
[509]	validation_0-auc:0.980264	validation_1-auc:0.933004
[510]	validation_0-auc:0.980276	validation_1-auc:0.93301
[511]	validation_0-auc:0.980261	validation_1-auc:0.933082
[512]	validation_0-auc:0.98027	validation_1-auc:0.933021
[513]	validation_0-auc:0.980311	validation_1-auc:0.932956
[514]	validation_0-auc:0.980343	validation_1-auc:0.932988
[515]	validation_0-auc:0.980367	validation_1-auc:0.932973
[516]	validation_0-auc:0.980369	validation_1-auc:0.933027
[517]	validation_0-auc:0.980403	validation_1-auc:0.933009
[518]	validation_0-auc:0.98044	validation_1-auc:0.933081
[519]	validation_0-auc:0.980533	validation_1-auc:0.932969
[520]	validation_0-auc:0.980589	validation_1-auc:0.932927
[521]	validation_0-auc:0.980597	validation_1-auc:0.932905
[522]	validation_0-auc:0.980594	validation_1-auc:0.932929
[523]	validation_0-auc:0.980593	validation_1-auc:0.932867
[524]	validation_0-auc:0.980644	validation_1-auc:0.932816
[525]	validation_0-auc:0.980647	validation_1-auc:0.932767
[526]	validation_0-auc:0.980687	validation_1-auc:0.932741
[527]	validation_0-auc:0.980714	validation_1-auc:0.932688
[528]	validation_0-auc:0.980763	validation_1-auc:0.932749
[529]	validation_0-auc:0.980823	validation_1-auc:0.932653
[530]	validation_0-auc:0.980874	validation_1-auc:0.932609
[531]	validation_0-auc:0.980888	validation_1-auc:0.932563
[532]	validation_0-auc:0.980893	validation_1-auc:0.932571
[533]	validation_0-auc:0.980921	validation_1-auc:0.932644
[534]	validation_0-auc:0.980999	validation_1-auc:0.932612
[535]	validation_0-auc:0.981032	validation_1-auc:0.932694
[536]	validation_0-auc:0.981048	validation_1-auc:0.932698
[537]	validation_0-auc:0.981053	validation_1-auc:0.932735
[538]	validation_0-auc:0.981101	validation_1-auc:0.932712
[539]	validation_0-auc:0.981083	validation_1-auc:0.932669
[540]	validation_0-auc:0.981102	validation_1-auc:0.932655
[541]	validation_0-auc:0.981159	validation_1-auc:0.932486
[542]	validation_0-auc:0.981138	validation_1-auc:0.932489
[543]	validation_0-auc:0.981177	validation_1-auc:0.932494
[544]	validation_0-auc:0.981251	validation_1-auc:0.932391
[545]	validation_0-auc:0.981226	validation_1-auc:0.932417
[546]	validation_0-auc:0.981317	validation_1-auc:0.932359
[547]	validation_0-auc:0.981343	validation_1-auc:0.932366
[548]	validation_0-auc:0.981405	validation_1-auc:0.932376
[549]	validation_0-auc:0.981439	validation_1-auc:0.932368
[550]	validation_0-auc:0.981446	validation_1-auc:0.9324
[551]	validation_0-auc:0.981512	validation_1-auc:0.932404
[552]	validation_0-auc:0.981542	validation_1-auc:0.932343
[553]	validation_0-auc:0.981566	validation_1-auc:0.932259
[554]	validation_0-auc:0.981578	validation_1-auc:0.932164
[555]	validation_0-auc:0.981593	validation_1-auc:0.932225
[556]	validation_0-auc:0.981569	validation_1-auc:0.932237
[557]	validation_0-auc:0.981549	validation_1-auc:0.932316
[558]	validation_0-auc:0.981567	validation_1-auc:0.932347
[559]	validation_0-auc:0.981591	validation_1-auc:0.932265
[560]	validation_0-auc:0.981589	validation_1-auc:0.932273
[561]	validation_0-auc:0.981607	validation_1-auc:0.932243
[562]	validation_0-auc:0.98162	validation_1-auc:0.932277
[563]	validation_0-auc:0.981646	validation_1-auc:0.932213
[564]	validation_0-auc:0.981687	validation_1-auc:0.932234
[565]	validation_0-auc:0.981743	validation_1-auc:0.932218
[566]	validation_0-auc:0.981773	validation_1-auc:0.932254
[567]	validation_0-auc:0.981834	validation_1-auc:0.932167
[568]	validation_0-auc:0.981846	validation_1-auc:0.932136
[569]	validation_0-auc:0.981903	validation_1-auc:0.932093
[570]	validation_0-auc:0.98195	validation_1-auc:0.932098
[571]	validation_0-auc:0.982019	validation_1-auc:0.932076
[572]	validation_0-auc:0.982013	validation_1-auc:0.932054
[573]	validation_0-auc:0.982074	validation_1-auc:0.93202
[574]	validation_0-auc:0.982105	validation_1-auc:0.932028
[575]	validation_0-auc:0.982093	validation_1-auc:0.932043
[576]	validation_0-auc:0.982118	validation_1-auc:0.932021
[577]	validation_0-auc:0.982122	validation_1-auc:0.932101
[578]	validation_0-auc:0.982149	validation_1-auc:0.932069
[579]	validation_0-auc:0.982212	validation_1-auc:0.932021
[580]	validation_0-auc:0.982223	validation_1-auc:0.932011
[581]	validation_0-auc:0.982294	validation_1-auc:0.932057
[582]	validation_0-auc:0.98231	validation_1-auc:0.932068
[583]	validation_0-auc:0.982332	validation_1-auc:0.932116
[584]	validation_0-auc:0.982381	validation_1-auc:0.932018
[585]	validation_0-auc:0.982406	validation_1-auc:0.931955
[586]	validation_0-auc:0.982455	validation_1-auc:0.93206
[587]	validation_0-auc:0.982457	validation_1-auc:0.932112
[588]	validation_0-auc:0.982429	validation_1-auc:0.932182
[589]	validation_0-auc:0.982484	validation_1-auc:0.932125
[590]	validation_0-auc:0.982478	validation_1-auc:0.93214
[591]	validation_0-auc:0.98248	validation_1-auc:0.932185
[592]	validation_0-auc:0.982478	validation_1-auc:0.932122
[593]	validation_0-auc:0.982492	validation_1-auc:0.932186
[594]	validation_0-auc:0.982546	validation_1-auc:0.932092
[595]	validation_0-auc:0.982594	validation_1-auc:0.93213
[596]	validation_0-auc:0.982588	validation_1-auc:0.932021
[597]	validation_0-auc:0.982591	validation_1-auc:0.931952
[598]	validation_0-auc:0.982595	validation_1-auc:0.932004
[599]	validation_0-auc:0.982655	validation_1-auc:0.931913
[600]	validation_0-auc:0.982697	validation_1-auc:0.93183
[601]	validation_0-auc:0.982712	validation_1-auc:0.931846
[602]	validation_0-auc:0.982736	validation_1-auc:0.931839
[603]	validation_0-auc:0.982746	validation_1-auc:0.931892
[604]	validation_0-auc:0.982772	validation_1-auc:0.931865
[605]	validation_0-auc:0.982805	validation_1-auc:0.931715
[606]	validation_0-auc:0.982888	validation_1-auc:0.931726
[607]	validation_0-auc:0.982993	validation_1-auc:0.931624
[608]	validation_0-auc:0.983008	validation_1-auc:0.931682
[609]	validation_0-auc:0.98302	validation_1-auc:0.931686
[610]	validation_0-auc:0.983102	validation_1-auc:0.931547
[611]	validation_0-auc:0.983101	validation_1-auc:0.93157
[612]	validation_0-auc:0.983072	validation_1-auc:0.931644
[613]	validation_0-auc:0.983097	validation_1-auc:0.931623
[614]	validation_0-auc:0.983155	validation_1-auc:0.931606
[615]	validation_0-auc:0.983144	validation_1-auc:0.931713
[616]	validation_0-auc:0.983178	validation_1-auc:0.931726
[617]	validation_0-auc:0.983216	validation_1-auc:0.931639
[618]	validation_0-auc:0.983261	validation_1-auc:0.931604
[619]	validation_0-auc:0.983257	validation_1-auc:0.931626
[620]	validation_0-auc:0.983266	validation_1-auc:0.931609
[621]	validation_0-auc:0.983309	validation_1-auc:0.931594
[622]	validation_0-auc:0.98332	validation_1-auc:0.931683
[623]	validation_0-auc:0.983327	validation_1-auc:0.931681
[624]	validation_0-auc:0.983363	validation_1-auc:0.931749
[625]	validation_0-auc:0.983391	validation_1-auc:0.931693
[626]	validation_0-auc:0.983404	validation_1-auc:0.931703
[627]	validation_0-auc:0.983397	validation_1-auc:0.931712
[628]	validation_0-auc:0.983418	validation_1-auc:0.931668
[629]	validation_0-auc:0.983456	validation_1-auc:0.931679
[630]	validation_0-auc:0.983467	validation_1-auc:0.93167
[631]	validation_0-auc:0.983485	validation_1-auc:0.931749
[632]	validation_0-auc:0.983511	validation_1-auc:0.931735
[633]	validation_0-auc:0.983554	validation_1-auc:0.931736
[634]	validation_0-auc:0.983566	validation_1-auc:0.931765
[635]	validation_0-auc:0.983578	validation_1-auc:0.931812
[636]	validation_0-auc:0.983583	validation_1-auc:0.93186
[637]	validation_0-auc:0.983595	validation_1-auc:0.931822
[638]	validation_0-auc:0.983643	validation_1-auc:0.931884
[639]	validation_0-auc:0.983663	validation_1-auc:0.931899
[640]	validation_0-auc:0.983696	validation_1-auc:0.931909
[641]	validation_0-auc:0.983705	validation_1-auc:0.931919
[642]	validation_0-auc:0.983747	validation_1-auc:0.931896
[643]	validation_0-auc:0.983781	validation_1-auc:0.931857
[644]	validation_0-auc:0.983802	validation_1-auc:0.931811
[645]	validation_0-auc:0.983854	validation_1-auc:0.931708
[646]	validation_0-auc:0.983896	validation_1-auc:0.931703
[647]	validation_0-auc:0.983945	validation_1-auc:0.931643
[648]	validation_0-auc:0.983971	validation_1-auc:0.931618
[649]	validation_0-auc:0.984033	validation_1-auc:0.93157
[650]	validation_0-auc:0.984047	validation_1-auc:0.931601
[651]	validation_0-auc:0.984094	validation_1-auc:0.931568
[652]	validation_0-auc:0.984135	validation_1-auc:0.931529
[653]	validation_0-auc:0.984136	validation_1-auc:0.931521
[654]	validation_0-auc:0.984159	validation_1-auc:0.931523
[655]	validation_0-auc:0.984162	validation_1-auc:0.931482
[656]	validation_0-auc:0.984179	validation_1-auc:0.931498
[657]	validation_0-auc:0.984266	validation_1-auc:0.931478
[658]	validation_0-auc:0.984336	validation_1-auc:0.931455
[659]	validation_0-auc:0.984366	validation_1-auc:0.931433
[660]	validation_0-auc:0.984429	validation_1-auc:0.931355
[661]	validation_0-auc:0.984467	validation_1-auc:0.931267
[662]	validation_0-auc:0.984469	validation_1-auc:0.931231
[663]	validation_0-auc:0.984513	validation_1-auc:0.931212
[664]	validation_0-auc:0.984516	validation_1-auc:0.931206
[665]	validation_0-auc:0.984567	validation_1-auc:0.931257
[666]	validation_0-auc:0.984607	validation_1-auc:0.931252
[667]	validation_0-auc:0.984686	validation_1-auc:0.9312
[668]	validation_0-auc:0.984703	validation_1-auc:0.93122
[669]	validation_0-auc:0.984762	validation_1-auc:0.931119
[670]	validation_0-auc:0.984798	validation_1-auc:0.931131
[671]	validation_0-auc:0.984799	validation_1-auc:0.931136
[672]	validation_0-auc:0.98481	validation_1-auc:0.931219
[673]	validation_0-auc:0.98483	validation_1-auc:0.931227
[674]	validation_0-auc:0.984822	validation_1-auc:0.931232
[675]	validation_0-auc:0.984826	validation_1-auc:0.931256
[676]	validation_0-auc:0.984823	validation_1-auc:0.931219
[677]	validation_0-auc:0.984892	validation_1-auc:0.931155
[678]	validation_0-auc:0.984918	validation_1-auc:0.93113
[679]	validation_0-auc:0.984928	validation_1-auc:0.931129
[680]	validation_0-auc:0.984903	validation_1-auc:0.931143
[681]	validation_0-auc:0.984905	validation_1-auc:0.931175
[682]	validation_0-auc:0.98491	validation_1-auc:0.931176
[683]	validation_0-auc:0.98491	validation_1-auc:0.931135
[684]	validation_0-auc:0.984918	validation_1-auc:0.931115
[685]	validation_0-auc:0.984955	validation_1-auc:0.931091
[686]	validation_0-auc:0.984953	validation_1-auc:0.931059
[687]	validation_0-auc:0.985001	validation_1-auc:0.931137
[688]	validation_0-auc:0.985012	validation_1-auc:0.931105
[689]	validation_0-auc:0.985019	validation_1-auc:0.931101
[690]	validation_0-auc:0.985015	validation_1-auc:0.931145
[691]	validation_0-auc:0.985048	validation_1-auc:0.931118
[692]	validation_0-auc:0.985069	validation_1-auc:0.931199
[693]	validation_0-auc:0.9851	validation_1-auc:0.931115
[694]	validation_0-auc:0.985082	validation_1-auc:0.931145
[695]	validation_0-auc:0.985092	validation_1-auc:0.931161
[696]	validation_0-auc:0.985145	validation_1-auc:0.931108
[697]	validation_0-auc:0.985176	validation_1-auc:0.931018
[698]	validation_0-auc:0.985234	validation_1-auc:0.930942
[699]	validation_0-auc:0.985219	validation_1-auc:0.930862
[700]	validation_0-auc:0.985269	validation_1-auc:0.930832
[701]	validation_0-auc:0.985291	validation_1-auc:0.930803
[702]	validation_0-auc:0.985313	validation_1-auc:0.930831
[703]	validation_0-auc:0.985369	validation_1-auc:0.930802
[704]	validation_0-auc:0.985405	validation_1-auc:0.930746
[705]	validation_0-auc:0.985413	validation_1-auc:0.930772
[706]	validation_0-auc:0.985446	validation_1-auc:0.930652
[707]	validation_0-auc:0.985498	validation_1-auc:0.930641
[708]	validation_0-auc:0.985491	validation_1-auc:0.930644
[709]	validation_0-auc:0.985514	validation_1-auc:0.930728
[710]	validation_0-auc:0.985496	validation_1-auc:0.930743
[711]	validation_0-auc:0.985507	validation_1-auc:0.930764
[712]	validation_0-auc:0.985564	validation_1-auc:0.930785
[713]	validation_0-auc:0.985575	validation_1-auc:0.930798
[714]	validation_0-auc:0.985641	validation_1-auc:0.93079
[715]	validation_0-auc:0.98571	validation_1-auc:0.93078
[716]	validation_0-auc:0.985752	validation_1-auc:0.930762
[717]	validation_0-auc:0.985753	validation_1-auc:0.930725
[718]	validation_0-auc:0.985786	validation_1-auc:0.930691
[719]	validation_0-auc:0.985802	validation_1-auc:0.930704
[720]	validation_0-auc:0.985785	validation_1-auc:0.930775
[721]	validation_0-auc:0.9858	validation_1-auc:0.930789
[722]	validation_0-auc:0.985827	validation_1-auc:0.930863
[723]	validation_0-auc:0.985858	validation_1-auc:0.930885
[724]	validation_0-auc:0.985901	validation_1-auc:0.93085
[725]	validation_0-auc:0.985901	validation_1-auc:0.930888
[726]	validation_0-auc:0.985961	validation_1-auc:0.930914
[727]	validation_0-auc:0.985999	validation_1-auc:0.930983
[728]	validation_0-auc:0.986015	validation_1-auc:0.930983
[729]	validation_0-auc:0.986039	validation_1-auc:0.930916
[730]	validation_0-auc:0.986046	validation_1-auc:0.931002
[731]	validation_0-auc:0.986064	validation_1-auc:0.931031
[732]	validation_0-auc:0.986101	validation_1-auc:0.931008
[733]	validation_0-auc:0.986117	validation_1-auc:0.931056
[734]	validation_0-auc:0.986078	validation_1-auc:0.931082
[735]	validation_0-auc:0.986112	validation_1-auc:0.930978
[736]	validation_0-auc:0.986102	validation_1-auc:0.931016
[737]	validation_0-auc:0.986153	validation_1-auc:0.931028
[738]	validation_0-auc:0.986186	validation_1-auc:0.930964
[739]	validation_0-auc:0.986181	validation_1-auc:0.930974
[740]	validation_0-auc:0.986229	validation_1-auc:0.930884
[741]	validation_0-auc:0.986322	validation_1-auc:0.930938
[742]	validation_0-auc:0.986317	validation_1-auc:0.930961
[743]	validation_0-auc:0.986339	validation_1-auc:0.93101
[744]	validation_0-auc:0.986393	validation_1-auc:0.930975
[745]	validation_0-auc:0.986416	validation_1-auc:0.930904
[746]	validation_0-auc:0.98643	validation_1-auc:0.930906
[747]	validation_0-auc:0.986437	validation_1-auc:0.930954
[748]	validation_0-auc:0.98646	validation_1-auc:0.930907
[749]	validation_0-auc:0.986531	validation_1-auc:0.930818
[750]	validation_0-auc:0.986527	validation_1-auc:0.930777
[751]	validation_0-auc:0.986527	validation_1-auc:0.930778
[752]	validation_0-auc:0.986515	validation_1-auc:0.930902
[753]	validation_0-auc:0.986535	validation_1-auc:0.930947
[754]	validation_0-auc:0.986532	validation_1-auc:0.930978
[755]	validation_0-auc:0.986582	validation_1-auc:0.93097
[756]	validation_0-auc:0.986593	validation_1-auc:0.930954
[757]	validation_0-auc:0.98662	validation_1-auc:0.930951
[758]	validation_0-auc:0.986652	validation_1-auc:0.930897
[759]	validation_0-auc:0.986662	validation_1-auc:0.930849
[760]	validation_0-auc:0.986677	validation_1-auc:0.930905
[761]	validation_0-auc:0.986655	validation_1-auc:0.930965
[762]	validation_0-auc:0.986688	validation_1-auc:0.930953
[763]	validation_0-auc:0.986718	validation_1-auc:0.930891
[764]	validation_0-auc:0.986715	validation_1-auc:0.930939
[765]	validation_0-auc:0.986721	validation_1-auc:0.9309
[766]	validation_0-auc:0.986742	validation_1-auc:0.930949
[767]	validation_0-auc:0.986781	validation_1-auc:0.930951
[768]	validation_0-auc:0.986796	validation_1-auc:0.93088
[769]	validation_0-auc:0.986817	validation_1-auc:0.930875
[770]	validation_0-auc:0.986844	validation_1-auc:0.930822
[771]	validation_0-auc:0.986878	validation_1-auc:0.930888
[772]	validation_0-auc:0.986901	validation_1-auc:0.930936
[773]	validation_0-auc:0.986896	validation_1-auc:0.931038
[774]	validation_0-auc:0.986915	validation_1-auc:0.931062
[775]	validation_0-auc:0.986935	validation_1-auc:0.931082
[776]	validation_0-auc:0.987007	validation_1-auc:0.930994
[777]	validation_0-auc:0.987071	validation_1-auc:0.93091
[778]	validation_0-auc:0.987035	validation_1-auc:0.930918
[779]	validation_0-auc:0.987051	validation_1-auc:0.930916
[780]	validation_0-auc:0.987071	validation_1-auc:0.930921
[781]	validation_0-auc:0.987073	validation_1-auc:0.930842
[782]	validation_0-auc:0.987096	validation_1-auc:0.930899
[783]	validation_0-auc:0.987107	validation_1-auc:0.930876
[784]	validation_0-auc:0.987107	validation_1-auc:0.930869
[785]	validation_0-auc:0.987132	validation_1-auc:0.930874
[786]	validation_0-auc:0.987143	validation_1-auc:0.930874
[787]	validation_0-auc:0.987162	validation_1-auc:0.930899
[788]	validation_0-auc:0.987187	validation_1-auc:0.930878
[789]	validation_0-auc:0.987199	validation_1-auc:0.930889
[790]	validation_0-auc:0.987216	validation_1-auc:0.93089
[791]	validation_0-auc:0.98724	validation_1-auc:0.930881
[792]	validation_0-auc:0.987253	validation_1-auc:0.930871
[793]	validation_0-auc:0.987288	validation_1-auc:0.930794
[794]	validation_0-auc:0.987299	validation_1-auc:0.930796
[795]	validation_0-auc:0.987332	validation_1-auc:0.930717
[796]	validation_0-auc:0.987347	validation_1-auc:0.93075
[797]	validation_0-auc:0.987355	validation_1-auc:0.930775
[798]	validation_0-auc:0.987369	validation_1-auc:0.930731
[799]	validation_0-auc:0.987423	validation_1-auc:0.930742
[800]	validation_0-auc:0.987461	validation_1-auc:0.93077
[801]	validation_0-auc:0.987473	validation_1-auc:0.930765
[802]	validation_0-auc:0.987505	validation_1-auc:0.930748
[803]	validation_0-auc:0.987493	validation_1-auc:0.930762
[804]	validation_0-auc:0.987476	validation_1-auc:0.930785
[805]	validation_0-auc:0.987498	validation_1-auc:0.930773
[806]	validation_0-auc:0.987523	validation_1-auc:0.930654
[807]	validation_0-auc:0.987574	validation_1-auc:0.930551
[808]	validation_0-auc:0.987607	validation_1-auc:0.930465
[809]	validation_0-auc:0.987596	validation_1-auc:0.930477
[810]	validation_0-auc:0.98762	validation_1-auc:0.930456
[811]	validation_0-auc:0.987631	validation_1-auc:0.930447
[812]	validation_0-auc:0.987614	validation_1-auc:0.930445
[813]	validation_0-auc:0.987631	validation_1-auc:0.930452
[814]	validation_0-auc:0.987648	validation_1-auc:0.930469
[815]	validation_0-auc:0.987626	validation_1-auc:0.930439
[816]	validation_0-auc:0.987654	validation_1-auc:0.93037
[817]	validation_0-auc:0.98764	validation_1-auc:0.93037
[818]	validation_0-auc:0.98764	validation_1-auc:0.930406
[819]	validation_0-auc:0.987652	validation_1-auc:0.93038
[820]	validation_0-auc:0.987627	validation_1-auc:0.930359
[821]	validation_0-auc:0.987643	validation_1-auc:0.930317
[822]	validation_0-auc:0.987681	validation_1-auc:0.930292
[823]	validation_0-auc:0.987676	validation_1-auc:0.930341
[824]	validation_0-auc:0.987707	validation_1-auc:0.930238
[825]	validation_0-auc:0.987719	validation_1-auc:0.930276
[826]	validation_0-auc:0.987723	validation_1-auc:0.930303
[827]	validation_0-auc:0.98775	validation_1-auc:0.930318
[828]	validation_0-auc:0.987738	validation_1-auc:0.930322
[829]	validation_0-auc:0.987707	validation_1-auc:0.930341
[830]	validation_0-auc:0.987743	validation_1-auc:0.93025
[831]	validation_0-auc:0.987763	validation_1-auc:0.930178
[832]	validation_0-auc:0.987815	validation_1-auc:0.93013
[833]	validation_0-auc:0.98781	validation_1-auc:0.930092
[834]	validation_0-auc:0.987824	validation_1-auc:0.930137
[835]	validation_0-auc:0.987823	validation_1-auc:0.930078
[836]	validation_0-auc:0.987863	validation_1-auc:0.930104
[837]	validation_0-auc:0.987885	validation_1-auc:0.930157
[838]	validation_0-auc:0.987932	validation_1-auc:0.930074
[839]	validation_0-auc:0.987949	validation_1-auc:0.930102
[840]	validation_0-auc:0.987955	validation_1-auc:0.93004
[841]	validation_0-auc:0.987968	validation_1-auc:0.929982
[842]	validation_0-auc:0.987998	validation_1-auc:0.929875
[843]	validation_0-auc:0.98802	validation_1-auc:0.929813
[844]	validation_0-auc:0.988044	validation_1-auc:0.92982
[845]	validation_0-auc:0.988054	validation_1-auc:0.929883
[846]	validation_0-auc:0.988123	validation_1-auc:0.9299
[847]	validation_0-auc:0.988185	validation_1-auc:0.92988
[848]	validation_0-auc:0.988213	validation_1-auc:0.929859
[849]	validation_0-auc:0.988237	validation_1-auc:0.929857
[850]	validation_0-auc:0.988259	validation_1-auc:0.929928
[851]	validation_0-auc:0.988267	validation_1-auc:0.929916
[852]	validation_0-auc:0.988319	validation_1-auc:0.929879
[853]	validation_0-auc:0.988335	validation_1-auc:0.929836
[854]	validation_0-auc:0.988356	validation_1-auc:0.929789
[855]	validation_0-auc:0.988365	validation_1-auc:0.929823
[856]	validation_0-auc:0.98835	validation_1-auc:0.92984
[857]	validation_0-auc:0.988366	validation_1-auc:0.929878
[858]	validation_0-auc:0.988382	validation_1-auc:0.929903
[859]	validation_0-auc:0.988388	validation_1-auc:0.929855
[860]	validation_0-auc:0.988439	validation_1-auc:0.929775
[861]	validation_0-auc:0.988442	validation_1-auc:0.929786
[862]	validation_0-auc:0.988463	validation_1-auc:0.92966
[863]	validation_0-auc:0.988508	validation_1-auc:0.929693
[864]	validation_0-auc:0.988524	validation_1-auc:0.929727
[865]	validation_0-auc:0.98853	validation_1-auc:0.929732
[866]	validation_0-auc:0.988582	validation_1-auc:0.929704
[867]	validation_0-auc:0.988621	validation_1-auc:0.929665
[868]	validation_0-auc:0.988611	validation_1-auc:0.929705
[869]	validation_0-auc:0.988578	validation_1-auc:0.929724
[870]	validation_0-auc:0.98858	validation_1-auc:0.929689
[871]	validation_0-auc:0.988607	validation_1-auc:0.929619
[872]	validation_0-auc:0.988593	validation_1-auc:0.92961
[873]	validation_0-auc:0.988608	validation_1-auc:0.929616
[874]	validation_0-auc:0.988608	validation_1-auc:0.929632
[875]	validation_0-auc:0.988651	validation_1-auc:0.929597
[876]	validation_0-auc:0.988643	validation_1-auc:0.929534
[877]	validation_0-auc:0.988653	validation_1-auc:0.929573
[878]	validation_0-auc:0.98869	validation_1-auc:0.929574
[879]	validation_0-auc:0.988687	validation_1-auc:0.929551
[880]	validation_0-auc:0.988697	validation_1-auc:0.929542
[881]	validation_0-auc:0.988745	validation_1-auc:0.929424
[882]	validation_0-auc:0.988727	validation_1-auc:0.929366
[883]	validation_0-auc:0.988743	validation_1-auc:0.929221
[884]	validation_0-auc:0.988776	validation_1-auc:0.929217
[885]	validation_0-auc:0.988776	validation_1-auc:0.929199
[886]	validation_0-auc:0.988833	validation_1-auc:0.929157
[887]	validation_0-auc:0.988837	validation_1-auc:0.929146
[888]	validation_0-auc:0.988846	validation_1-auc:0.929177
[889]	validation_0-auc:0.988857	validation_1-auc:0.929175
[890]	validation_0-auc:0.988859	validation_1-auc:0.929215
[891]	validation_0-auc:0.988855	validation_1-auc:0.929228
[892]	validation_0-auc:0.988873	validation_1-auc:0.92921
[893]	validation_0-auc:0.988926	validation_1-auc:0.929172
[894]	validation_0-auc:0.988939	validation_1-auc:0.929107
[895]	validation_0-auc:0.988951	validation_1-auc:0.92909
[896]	validation_0-auc:0.988977	validation_1-auc:0.92905
[897]	validation_0-auc:0.989027	validation_1-auc:0.929051
[898]	validation_0-auc:0.989026	validation_1-auc:0.929091
[899]	validation_0-auc:0.989055	validation_1-auc:0.929034
[900]	validation_0-auc:0.989072	validation_1-auc:0.929086
[901]	validation_0-auc:0.989081	validation_1-auc:0.929121
[902]	validation_0-auc:0.989057	validation_1-auc:0.929123
[903]	validation_0-auc:0.989056	validation_1-auc:0.929168
[904]	validation_0-auc:0.989092	validation_1-auc:0.929106
[905]	validation_0-auc:0.989102	validation_1-auc:0.929123
[906]	validation_0-auc:0.989112	validation_1-auc:0.929066
[907]	validation_0-auc:0.98912	validation_1-auc:0.929079
[908]	validation_0-auc:0.9891	validation_1-auc:0.92916
[909]	validation_0-auc:0.9891	validation_1-auc:0.929193
[910]	validation_0-auc:0.989118	validation_1-auc:0.929175
[911]	validation_0-auc:0.989136	validation_1-auc:0.929146
[912]	validation_0-auc:0.989158	validation_1-auc:0.929204
[913]	validation_0-auc:0.989218	validation_1-auc:0.929242
[914]	validation_0-auc:0.98923	validation_1-auc:0.929245
[915]	validation_0-auc:0.989231	validation_1-auc:0.92923
[916]	validation_0-auc:0.989269	validation_1-auc:0.929198
[917]	validation_0-auc:0.989307	validation_1-auc:0.929091
[918]	validation_0-auc:0.989315	validation_1-auc:0.92909
[919]	validation_0-auc:0.989314	validation_1-auc:0.929162
[920]	validation_0-auc:0.989331	validation_1-auc:0.929212
[921]	validation_0-auc:0.989344	validation_1-auc:0.929121
[922]	validation_0-auc:0.989344	validation_1-auc:0.929161
[923]	validation_0-auc:0.989381	validation_1-auc:0.929161
[924]	validation_0-auc:0.989394	validation_1-auc:0.929066
[925]	validation_0-auc:0.989401	validation_1-auc:0.929111
[926]	validation_0-auc:0.98942	validation_1-auc:0.929109
[927]	validation_0-auc:0.989427	validation_1-auc:0.929136
[928]	validation_0-auc:0.989465	validation_1-auc:0.929109
[929]	validation_0-auc:0.989472	validation_1-auc:0.929096
[930]	validation_0-auc:0.989534	validation_1-auc:0.929011
[931]	validation_0-auc:0.989526	validation_1-auc:0.92891
[932]	validation_0-auc:0.989531	validation_1-auc:0.928968
[933]	validation_0-auc:0.98954	validation_1-auc:0.929016
[934]	validation_0-auc:0.989565	validation_1-auc:0.929009
[935]	validation_0-auc:0.98956	validation_1-auc:0.928976
[936]	validation_0-auc:0.989642	validation_1-auc:0.928829
[937]	validation_0-auc:0.989628	validation_1-auc:0.928858
[938]	validation_0-auc:0.989648	validation_1-auc:0.928793
[939]	validation_0-auc:0.989687	validation_1-auc:0.928756
[940]	validation_0-auc:0.989741	validation_1-auc:0.928727
[941]	validation_0-auc:0.989743	validation_1-auc:0.928751
[942]	validation_0-auc:0.98976	validation_1-auc:0.928797
[943]	validation_0-auc:0.989807	validation_1-auc:0.928761
[944]	validation_0-auc:0.989808	validation_1-auc:0.92878
[945]	validation_0-auc:0.98983	validation_1-auc:0.928776
[946]	validation_0-auc:0.989814	validation_1-auc:0.928834
[947]	validation_0-auc:0.989838	validation_1-auc:0.928859
[948]	validation_0-auc:0.989852	validation_1-auc:0.928848
[949]	validation_0-auc:0.989867	validation_1-auc:0.928871
[950]	validation_0-auc:0.989862	validation_1-auc:0.928879
[951]	validation_0-auc:0.989845	validation_1-auc:0.928828
[952]	validation_0-auc:0.989876	validation_1-auc:0.928815
[953]	validation_0-auc:0.989917	validation_1-auc:0.928715
[954]	validation_0-auc:0.989962	validation_1-auc:0.928663
[955]	validation_0-auc:0.989958	validation_1-auc:0.928717
[956]	validation_0-auc:0.989953	validation_1-auc:0.928702
[957]	validation_0-auc:0.989933	validation_1-auc:0.928769
[958]	validation_0-auc:0.989959	validation_1-auc:0.928744
[959]	validation_0-auc:0.989953	validation_1-auc:0.928819
[960]	validation_0-auc:0.989983	validation_1-auc:0.928768
[961]	validation_0-auc:0.989984	validation_1-auc:0.92881
[962]	validation_0-auc:0.989974	validation_1-auc:0.928847
[963]	validation_0-auc:0.990031	validation_1-auc:0.928723
[964]	validation_0-auc:0.990029	validation_1-auc:0.928783
[965]	validation_0-auc:0.990053	validation_1-auc:0.928788
[966]	validation_0-auc:0.990066	validation_1-auc:0.928857
[967]	validation_0-auc:0.990095	validation_1-auc:0.928894
[968]	validation_0-auc:0.990092	validation_1-auc:0.928899
[969]	validation_0-auc:0.9901	validation_1-auc:0.928878
[970]	validation_0-auc:0.990105	validation_1-auc:0.928867
[971]	validation_0-auc:0.990119	validation_1-auc:0.928837
[972]	validation_0-auc:0.990134	validation_1-auc:0.928846
[973]	validation_0-auc:0.990146	validation_1-auc:0.928822
[974]	validation_0-auc:0.990165	validation_1-auc:0.928755
[975]	validation_0-auc:0.990163	validation_1-auc:0.928741
[976]	validation_0-auc:0.99019	validation_1-auc:0.928757
[977]	validation_0-auc:0.990204	validation_1-auc:0.928712
[978]	validation_0-auc:0.990233	validation_1-auc:0.928694
[979]	validation_0-auc:0.990246	validation_1-auc:0.928686
[980]	validation_0-auc:0.990279	validation_1-auc:0.92872
[981]	validation_0-auc:0.990296	validation_1-auc:0.928744
[982]	validation_0-auc:0.990333	validation_1-auc:0.928771
[983]	validation_0-auc:0.990354	validation_1-auc:0.928653
[984]	validation_0-auc:0.990361	validation_1-auc:0.928666
[985]	validation_0-auc:0.99041	validation_1-auc:0.928576
[986]	validation_0-auc:0.990443	validation_1-auc:0.928576
[987]	validation_0-auc:0.990454	validation_1-auc:0.928594
[988]	validation_0-auc:0.990458	validation_1-auc:0.928563
[989]	validation_0-auc:0.990489	validation_1-auc:0.928532
[990]	validation_0-auc:0.990489	validation_1-auc:0.928521
[991]	validation_0-auc:0.990485	validation_1-auc:0.928546
[992]	validation_0-auc:0.990489	validation_1-auc:0.928542
[993]	validation_0-auc:0.990481	validation_1-auc:0.928541
[994]	validation_0-auc:0.990482	validation_1-auc:0.928563
[995]	validation_0-auc:0.990488	validation_1-auc:0.928546
[996]	validation_0-auc:0.990474	validation_1-auc:0.928641
[997]	validation_0-auc:0.990503	validation_1-auc:0.928528
[998]	validation_0-auc:0.990515	validation_1-auc:0.928467
[999]	validation_0-auc:0.990556	validation_1-auc:0.928361
Wall time: 14 s
Out[97]:
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1,
              disable_default_eval_metric=1, eval_metric='auc', gamma=0.15,
              learning_rate=0.1, max_delta_step=0, max_depth=7,
              min_child_weight=3, missing=None, n_estimators=1000, n_jobs=8,
              nthread=None, objective='binary:logistic', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=7.8243170862346005,
              seed=None, silent=None, subsample=0.5, verbose_eval=100,
              verbosity=0)
In [98]:
true_flag = test_df_total['y']
pred_flag = clf2.predict(test_df_total[top_5_feat])

print("AUC score: {:<8.3f}".format(roc_auc_score(true_flag, pred_flag)))
AUC score: 0.830   

Check confusion matrix, AUC plot and other metrics

In [99]:
metrics = confusion_mat_plot(true_flag, pred_flag)
accuracy: 87.96 %
precision: 47.09 %
Recall: 76.71 %
specificity: 89.35 %
F-score: 58.35 %
In [100]:
gini = auc_plot(test_df_total[top_5_feat], true_flag, clf2)

# record new metrics
model_comparison['With Eng Feat-Top 5'] = [gini, metrics[0], metrics[1], metrics[2], metrics[3], metrics[4]]
gini 85.672 %

Here we can clearly see that we are losing in performance by a little in our simplified model, but we are having a less complicated model.

Let's check again the potential changes in feature importance (with engineered features).

In [101]:
feature_importance_df2 = pd.DataFrame()
feature_importance_df2["Feature"] = top_5_feat
feature_importance_df2["importance"] = clf2.feature_importances_

# plot results
plt.figure(figsize=(12,12))
sns.barplot(x="importance", y="Feature", data=feature_importance_df2.sort_values(by="importance",ascending=False))
plt.title('Features importance')
plt.tight_layout()
plt.show()

We will now try to use only the original processed data, without the engineered features.

Model 3 - No engineered features

In [102]:
%%time
clf3 = xgb.XGBClassifier(**best_parameters)
tr_val = (train_df_total[train_df_total.columns[:-9]], target)
ts_val = (test_df_total[train_df_total.columns[:-9]], test_df_total['y'])

clf3.fit(train_df_total[train_df_total.columns[:-9]], target, eval_metric='auc', eval_set=[tr_val, ts_val])
[0]	validation_0-auc:0.943149	validation_1-auc:0.933155
[1]	validation_0-auc:0.949653	validation_1-auc:0.938549
[2]	validation_0-auc:0.951773	validation_1-auc:0.940949
[3]	validation_0-auc:0.952958	validation_1-auc:0.94173
[4]	validation_0-auc:0.954174	validation_1-auc:0.942741
[5]	validation_0-auc:0.955023	validation_1-auc:0.943218
[6]	validation_0-auc:0.955586	validation_1-auc:0.943614
[7]	validation_0-auc:0.956291	validation_1-auc:0.943741
[8]	validation_0-auc:0.956568	validation_1-auc:0.94401
[9]	validation_0-auc:0.95712	validation_1-auc:0.944055
[10]	validation_0-auc:0.957835	validation_1-auc:0.944332
[11]	validation_0-auc:0.958249	validation_1-auc:0.944745
[12]	validation_0-auc:0.958522	validation_1-auc:0.944726
[13]	validation_0-auc:0.958976	validation_1-auc:0.94432
[14]	validation_0-auc:0.959321	validation_1-auc:0.94428
[15]	validation_0-auc:0.959604	validation_1-auc:0.944465
[16]	validation_0-auc:0.959926	validation_1-auc:0.944553
[17]	validation_0-auc:0.96005	validation_1-auc:0.944526
[18]	validation_0-auc:0.960442	validation_1-auc:0.944524
[19]	validation_0-auc:0.960841	validation_1-auc:0.945173
[20]	validation_0-auc:0.961041	validation_1-auc:0.945326
[21]	validation_0-auc:0.961259	validation_1-auc:0.94536
[22]	validation_0-auc:0.961513	validation_1-auc:0.945453
[23]	validation_0-auc:0.961707	validation_1-auc:0.945517
[24]	validation_0-auc:0.962018	validation_1-auc:0.945924
[25]	validation_0-auc:0.962263	validation_1-auc:0.946021
[26]	validation_0-auc:0.962462	validation_1-auc:0.946022
[27]	validation_0-auc:0.9627	validation_1-auc:0.946183
[28]	validation_0-auc:0.962987	validation_1-auc:0.946121
[29]	validation_0-auc:0.963146	validation_1-auc:0.946152
[30]	validation_0-auc:0.963297	validation_1-auc:0.946231
[31]	validation_0-auc:0.963468	validation_1-auc:0.946347
[32]	validation_0-auc:0.963577	validation_1-auc:0.94643
[33]	validation_0-auc:0.963887	validation_1-auc:0.946446
[34]	validation_0-auc:0.964251	validation_1-auc:0.946563
[35]	validation_0-auc:0.964335	validation_1-auc:0.946509
[36]	validation_0-auc:0.964494	validation_1-auc:0.946401
[37]	validation_0-auc:0.96468	validation_1-auc:0.946269
[38]	validation_0-auc:0.964844	validation_1-auc:0.946408
[39]	validation_0-auc:0.965136	validation_1-auc:0.946488
[40]	validation_0-auc:0.965351	validation_1-auc:0.946606
[41]	validation_0-auc:0.96544	validation_1-auc:0.946723
[42]	validation_0-auc:0.965681	validation_1-auc:0.946755
[43]	validation_0-auc:0.965925	validation_1-auc:0.946714
[44]	validation_0-auc:0.966093	validation_1-auc:0.946579
[45]	validation_0-auc:0.96616	validation_1-auc:0.9465
[46]	validation_0-auc:0.966279	validation_1-auc:0.946491
[47]	validation_0-auc:0.966514	validation_1-auc:0.946398
[48]	validation_0-auc:0.966749	validation_1-auc:0.946403
[49]	validation_0-auc:0.967076	validation_1-auc:0.946519
[50]	validation_0-auc:0.967381	validation_1-auc:0.946532
[51]	validation_0-auc:0.967552	validation_1-auc:0.946341
[52]	validation_0-auc:0.967698	validation_1-auc:0.946422
[53]	validation_0-auc:0.967986	validation_1-auc:0.946381
[54]	validation_0-auc:0.96813	validation_1-auc:0.946377
[55]	validation_0-auc:0.968297	validation_1-auc:0.946187
[56]	validation_0-auc:0.968473	validation_1-auc:0.946159
[57]	validation_0-auc:0.968671	validation_1-auc:0.94609
[58]	validation_0-auc:0.96885	validation_1-auc:0.94598
[59]	validation_0-auc:0.969064	validation_1-auc:0.945972
[60]	validation_0-auc:0.96929	validation_1-auc:0.945876
[61]	validation_0-auc:0.969544	validation_1-auc:0.945906
[62]	validation_0-auc:0.969838	validation_1-auc:0.9459
[63]	validation_0-auc:0.970057	validation_1-auc:0.945876
[64]	validation_0-auc:0.970212	validation_1-auc:0.945805
[65]	validation_0-auc:0.970607	validation_1-auc:0.945738
[66]	validation_0-auc:0.970723	validation_1-auc:0.945925
[67]	validation_0-auc:0.970898	validation_1-auc:0.945958
[68]	validation_0-auc:0.971002	validation_1-auc:0.945947
[69]	validation_0-auc:0.971301	validation_1-auc:0.945883
[70]	validation_0-auc:0.971524	validation_1-auc:0.945913
[71]	validation_0-auc:0.971773	validation_1-auc:0.945911
[72]	validation_0-auc:0.971936	validation_1-auc:0.945822
[73]	validation_0-auc:0.972121	validation_1-auc:0.945856
[74]	validation_0-auc:0.972216	validation_1-auc:0.945796
[75]	validation_0-auc:0.972412	validation_1-auc:0.945875
[76]	validation_0-auc:0.972515	validation_1-auc:0.945885
[77]	validation_0-auc:0.972698	validation_1-auc:0.945826
[78]	validation_0-auc:0.972952	validation_1-auc:0.945723
[79]	validation_0-auc:0.973212	validation_1-auc:0.945669
[80]	validation_0-auc:0.97339	validation_1-auc:0.945653
[81]	validation_0-auc:0.973568	validation_1-auc:0.945748
[82]	validation_0-auc:0.973766	validation_1-auc:0.945823
[83]	validation_0-auc:0.973963	validation_1-auc:0.945707
[84]	validation_0-auc:0.974256	validation_1-auc:0.945534
[85]	validation_0-auc:0.97463	validation_1-auc:0.945305
[86]	validation_0-auc:0.974887	validation_1-auc:0.94527
[87]	validation_0-auc:0.975115	validation_1-auc:0.945307
[88]	validation_0-auc:0.975303	validation_1-auc:0.945342
[89]	validation_0-auc:0.975421	validation_1-auc:0.945363
[90]	validation_0-auc:0.975488	validation_1-auc:0.945424
[91]	validation_0-auc:0.975726	validation_1-auc:0.945459
[92]	validation_0-auc:0.975887	validation_1-auc:0.945283
[93]	validation_0-auc:0.976088	validation_1-auc:0.945096
[94]	validation_0-auc:0.976248	validation_1-auc:0.945084
[95]	validation_0-auc:0.976399	validation_1-auc:0.94514
[96]	validation_0-auc:0.976471	validation_1-auc:0.945066
[97]	validation_0-auc:0.976622	validation_1-auc:0.94507
[98]	validation_0-auc:0.976718	validation_1-auc:0.944977
[99]	validation_0-auc:0.977048	validation_1-auc:0.944947
[100]	validation_0-auc:0.977189	validation_1-auc:0.945078
[101]	validation_0-auc:0.977368	validation_1-auc:0.945155
[102]	validation_0-auc:0.977639	validation_1-auc:0.94498
[103]	validation_0-auc:0.977803	validation_1-auc:0.944976
[104]	validation_0-auc:0.977974	validation_1-auc:0.944966
[105]	validation_0-auc:0.978062	validation_1-auc:0.944966
[106]	validation_0-auc:0.978198	validation_1-auc:0.945001
[107]	validation_0-auc:0.978256	validation_1-auc:0.945032
[108]	validation_0-auc:0.978348	validation_1-auc:0.944973
[109]	validation_0-auc:0.978516	validation_1-auc:0.944953
[110]	validation_0-auc:0.978674	validation_1-auc:0.944913
[111]	validation_0-auc:0.978768	validation_1-auc:0.944916
[112]	validation_0-auc:0.978889	validation_1-auc:0.944939
[113]	validation_0-auc:0.979004	validation_1-auc:0.944906
[114]	validation_0-auc:0.979161	validation_1-auc:0.944927
[115]	validation_0-auc:0.979475	validation_1-auc:0.944875
[116]	validation_0-auc:0.979592	validation_1-auc:0.944892
[117]	validation_0-auc:0.979666	validation_1-auc:0.944861
[118]	validation_0-auc:0.979871	validation_1-auc:0.944694
[119]	validation_0-auc:0.98015	validation_1-auc:0.944564
[120]	validation_0-auc:0.980289	validation_1-auc:0.94447
[121]	validation_0-auc:0.980506	validation_1-auc:0.944602
[122]	validation_0-auc:0.980602	validation_1-auc:0.94451
[123]	validation_0-auc:0.980691	validation_1-auc:0.944464
[124]	validation_0-auc:0.980794	validation_1-auc:0.944437
[125]	validation_0-auc:0.980941	validation_1-auc:0.944478
[126]	validation_0-auc:0.981058	validation_1-auc:0.944557
[127]	validation_0-auc:0.981081	validation_1-auc:0.944627
[128]	validation_0-auc:0.981176	validation_1-auc:0.944537
[129]	validation_0-auc:0.981269	validation_1-auc:0.944539
[130]	validation_0-auc:0.981372	validation_1-auc:0.944476
[131]	validation_0-auc:0.981522	validation_1-auc:0.944398
[132]	validation_0-auc:0.981665	validation_1-auc:0.944527
[133]	validation_0-auc:0.981834	validation_1-auc:0.944582
[134]	validation_0-auc:0.981918	validation_1-auc:0.944612
[135]	validation_0-auc:0.981959	validation_1-auc:0.944609
[136]	validation_0-auc:0.982111	validation_1-auc:0.944578
[137]	validation_0-auc:0.982261	validation_1-auc:0.944518
[138]	validation_0-auc:0.982322	validation_1-auc:0.944476
[139]	validation_0-auc:0.982498	validation_1-auc:0.944419
[140]	validation_0-auc:0.982748	validation_1-auc:0.944311
[141]	validation_0-auc:0.982811	validation_1-auc:0.94435
[142]	validation_0-auc:0.982934	validation_1-auc:0.944425
[143]	validation_0-auc:0.983015	validation_1-auc:0.944407
[144]	validation_0-auc:0.98319	validation_1-auc:0.944339
[145]	validation_0-auc:0.983314	validation_1-auc:0.944156
[146]	validation_0-auc:0.983413	validation_1-auc:0.944119
[147]	validation_0-auc:0.983532	validation_1-auc:0.944137
[148]	validation_0-auc:0.983623	validation_1-auc:0.944151
[149]	validation_0-auc:0.98363	validation_1-auc:0.944273
[150]	validation_0-auc:0.983801	validation_1-auc:0.944172
[151]	validation_0-auc:0.983838	validation_1-auc:0.944292
[152]	validation_0-auc:0.983936	validation_1-auc:0.944235
[153]	validation_0-auc:0.984066	validation_1-auc:0.944235
[154]	validation_0-auc:0.984084	validation_1-auc:0.944201
[155]	validation_0-auc:0.984208	validation_1-auc:0.944183
[156]	validation_0-auc:0.984309	validation_1-auc:0.944135
[157]	validation_0-auc:0.98443	validation_1-auc:0.944165
[158]	validation_0-auc:0.984572	validation_1-auc:0.944153
[159]	validation_0-auc:0.984659	validation_1-auc:0.944071
[160]	validation_0-auc:0.984728	validation_1-auc:0.943933
[161]	validation_0-auc:0.984813	validation_1-auc:0.943818
[162]	validation_0-auc:0.984877	validation_1-auc:0.943835
[163]	validation_0-auc:0.984965	validation_1-auc:0.943805
[164]	validation_0-auc:0.985038	validation_1-auc:0.943784
[165]	validation_0-auc:0.985253	validation_1-auc:0.943812
[166]	validation_0-auc:0.9853	validation_1-auc:0.943778
[167]	validation_0-auc:0.985433	validation_1-auc:0.943563
[168]	validation_0-auc:0.985545	validation_1-auc:0.943621
[169]	validation_0-auc:0.985667	validation_1-auc:0.943548
[170]	validation_0-auc:0.985846	validation_1-auc:0.943457
[171]	validation_0-auc:0.985933	validation_1-auc:0.943435
[172]	validation_0-auc:0.986014	validation_1-auc:0.943381
[173]	validation_0-auc:0.986026	validation_1-auc:0.943367
[174]	validation_0-auc:0.986145	validation_1-auc:0.943312
[175]	validation_0-auc:0.986178	validation_1-auc:0.943219
[176]	validation_0-auc:0.986285	validation_1-auc:0.943286
[177]	validation_0-auc:0.986345	validation_1-auc:0.943353
[178]	validation_0-auc:0.986402	validation_1-auc:0.9434
[179]	validation_0-auc:0.986537	validation_1-auc:0.943317
[180]	validation_0-auc:0.986581	validation_1-auc:0.943358
[181]	validation_0-auc:0.986612	validation_1-auc:0.943377
[182]	validation_0-auc:0.986659	validation_1-auc:0.94339
[183]	validation_0-auc:0.986791	validation_1-auc:0.943321
[184]	validation_0-auc:0.98688	validation_1-auc:0.943343
[185]	validation_0-auc:0.987033	validation_1-auc:0.943278
[186]	validation_0-auc:0.987115	validation_1-auc:0.943253
[187]	validation_0-auc:0.987244	validation_1-auc:0.943327
[188]	validation_0-auc:0.987285	validation_1-auc:0.943342
[189]	validation_0-auc:0.987403	validation_1-auc:0.943268
[190]	validation_0-auc:0.987539	validation_1-auc:0.943274
[191]	validation_0-auc:0.987612	validation_1-auc:0.943345
[192]	validation_0-auc:0.987665	validation_1-auc:0.943337
[193]	validation_0-auc:0.987768	validation_1-auc:0.943424
[194]	validation_0-auc:0.987885	validation_1-auc:0.943398
[195]	validation_0-auc:0.988008	validation_1-auc:0.943413
[196]	validation_0-auc:0.988036	validation_1-auc:0.94336
[197]	validation_0-auc:0.98813	validation_1-auc:0.943344
[198]	validation_0-auc:0.988236	validation_1-auc:0.943257
[199]	validation_0-auc:0.98834	validation_1-auc:0.943231
[200]	validation_0-auc:0.988424	validation_1-auc:0.943181
[201]	validation_0-auc:0.988513	validation_1-auc:0.943165
[202]	validation_0-auc:0.988588	validation_1-auc:0.943117
[203]	validation_0-auc:0.988733	validation_1-auc:0.94308
[204]	validation_0-auc:0.9888	validation_1-auc:0.943027
[205]	validation_0-auc:0.988926	validation_1-auc:0.942947
[206]	validation_0-auc:0.989011	validation_1-auc:0.942938
[207]	validation_0-auc:0.989068	validation_1-auc:0.942924
[208]	validation_0-auc:0.989155	validation_1-auc:0.942864
[209]	validation_0-auc:0.989251	validation_1-auc:0.942806
[210]	validation_0-auc:0.98931	validation_1-auc:0.942801
[211]	validation_0-auc:0.989367	validation_1-auc:0.94285
[212]	validation_0-auc:0.989479	validation_1-auc:0.942735
[213]	validation_0-auc:0.989556	validation_1-auc:0.942746
[214]	validation_0-auc:0.989692	validation_1-auc:0.942631
[215]	validation_0-auc:0.989817	validation_1-auc:0.942641
[216]	validation_0-auc:0.989858	validation_1-auc:0.942662
[217]	validation_0-auc:0.989878	validation_1-auc:0.942682
[218]	validation_0-auc:0.989935	validation_1-auc:0.942698
[219]	validation_0-auc:0.989968	validation_1-auc:0.94269
[220]	validation_0-auc:0.990008	validation_1-auc:0.942634
[221]	validation_0-auc:0.990125	validation_1-auc:0.942629
[222]	validation_0-auc:0.99019	validation_1-auc:0.94253
[223]	validation_0-auc:0.990236	validation_1-auc:0.942573
[224]	validation_0-auc:0.990297	validation_1-auc:0.942534
[225]	validation_0-auc:0.99034	validation_1-auc:0.942505
[226]	validation_0-auc:0.990427	validation_1-auc:0.942464
[227]	validation_0-auc:0.990454	validation_1-auc:0.942439
[228]	validation_0-auc:0.990559	validation_1-auc:0.942452
[229]	validation_0-auc:0.990628	validation_1-auc:0.942419
[230]	validation_0-auc:0.990705	validation_1-auc:0.942423
[231]	validation_0-auc:0.990748	validation_1-auc:0.942376
[232]	validation_0-auc:0.990808	validation_1-auc:0.942424
[233]	validation_0-auc:0.990893	validation_1-auc:0.942476
[234]	validation_0-auc:0.99094	validation_1-auc:0.942477
[235]	validation_0-auc:0.991095	validation_1-auc:0.94243
[236]	validation_0-auc:0.991157	validation_1-auc:0.942333
[237]	validation_0-auc:0.99123	validation_1-auc:0.942234
[238]	validation_0-auc:0.99137	validation_1-auc:0.94218
[239]	validation_0-auc:0.991444	validation_1-auc:0.94214
[240]	validation_0-auc:0.991517	validation_1-auc:0.94198
[241]	validation_0-auc:0.991552	validation_1-auc:0.941981
[242]	validation_0-auc:0.991631	validation_1-auc:0.941985
[243]	validation_0-auc:0.99174	validation_1-auc:0.94186
[244]	validation_0-auc:0.991823	validation_1-auc:0.941745
[245]	validation_0-auc:0.991879	validation_1-auc:0.941787
[246]	validation_0-auc:0.991951	validation_1-auc:0.941855
[247]	validation_0-auc:0.991975	validation_1-auc:0.941902
[248]	validation_0-auc:0.992023	validation_1-auc:0.941804
[249]	validation_0-auc:0.992097	validation_1-auc:0.941748
[250]	validation_0-auc:0.992164	validation_1-auc:0.941782
[251]	validation_0-auc:0.9922	validation_1-auc:0.941867
[252]	validation_0-auc:0.992231	validation_1-auc:0.941893
[253]	validation_0-auc:0.992276	validation_1-auc:0.941925
[254]	validation_0-auc:0.992315	validation_1-auc:0.941922
[255]	validation_0-auc:0.992362	validation_1-auc:0.941961
[256]	validation_0-auc:0.992415	validation_1-auc:0.941814
[257]	validation_0-auc:0.992458	validation_1-auc:0.941826
[258]	validation_0-auc:0.99248	validation_1-auc:0.941754
[259]	validation_0-auc:0.992546	validation_1-auc:0.941779
[260]	validation_0-auc:0.992607	validation_1-auc:0.941851
[261]	validation_0-auc:0.992612	validation_1-auc:0.941935
[262]	validation_0-auc:0.992639	validation_1-auc:0.941917
[263]	validation_0-auc:0.992673	validation_1-auc:0.941998
[264]	validation_0-auc:0.992726	validation_1-auc:0.941901
[265]	validation_0-auc:0.992772	validation_1-auc:0.941927
[266]	validation_0-auc:0.992817	validation_1-auc:0.941829
[267]	validation_0-auc:0.992877	validation_1-auc:0.941823
[268]	validation_0-auc:0.9929	validation_1-auc:0.941876
[269]	validation_0-auc:0.992985	validation_1-auc:0.941836
[270]	validation_0-auc:0.99303	validation_1-auc:0.941859
[271]	validation_0-auc:0.993105	validation_1-auc:0.941811
[272]	validation_0-auc:0.993154	validation_1-auc:0.941762
[273]	validation_0-auc:0.993187	validation_1-auc:0.941761
[274]	validation_0-auc:0.993222	validation_1-auc:0.9418
[275]	validation_0-auc:0.99329	validation_1-auc:0.941735
[276]	validation_0-auc:0.993328	validation_1-auc:0.941626
[277]	validation_0-auc:0.993398	validation_1-auc:0.941567
[278]	validation_0-auc:0.993451	validation_1-auc:0.941544
[279]	validation_0-auc:0.993449	validation_1-auc:0.941555
[280]	validation_0-auc:0.993476	validation_1-auc:0.941676
[281]	validation_0-auc:0.993557	validation_1-auc:0.941711
[282]	validation_0-auc:0.993579	validation_1-auc:0.941629
[283]	validation_0-auc:0.993605	validation_1-auc:0.941661
[284]	validation_0-auc:0.993627	validation_1-auc:0.941702
[285]	validation_0-auc:0.993675	validation_1-auc:0.941656
[286]	validation_0-auc:0.993705	validation_1-auc:0.94166
[287]	validation_0-auc:0.993736	validation_1-auc:0.941587
[288]	validation_0-auc:0.993779	validation_1-auc:0.941712
[289]	validation_0-auc:0.993809	validation_1-auc:0.94177
[290]	validation_0-auc:0.993819	validation_1-auc:0.94183
[291]	validation_0-auc:0.993928	validation_1-auc:0.941818
[292]	validation_0-auc:0.993975	validation_1-auc:0.941736
[293]	validation_0-auc:0.994029	validation_1-auc:0.941688
[294]	validation_0-auc:0.994057	validation_1-auc:0.941635
[295]	validation_0-auc:0.994109	validation_1-auc:0.941615
[296]	validation_0-auc:0.994157	validation_1-auc:0.941614
[297]	validation_0-auc:0.994186	validation_1-auc:0.941568
[298]	validation_0-auc:0.994203	validation_1-auc:0.941587
[299]	validation_0-auc:0.994247	validation_1-auc:0.941524
[300]	validation_0-auc:0.994311	validation_1-auc:0.941443
[301]	validation_0-auc:0.994394	validation_1-auc:0.941454
[302]	validation_0-auc:0.99441	validation_1-auc:0.941537
[303]	validation_0-auc:0.994429	validation_1-auc:0.941527
[304]	validation_0-auc:0.994515	validation_1-auc:0.941581
[305]	validation_0-auc:0.994546	validation_1-auc:0.941492
[306]	validation_0-auc:0.994555	validation_1-auc:0.941495
[307]	validation_0-auc:0.994598	validation_1-auc:0.941484
[308]	validation_0-auc:0.994601	validation_1-auc:0.941535
[309]	validation_0-auc:0.994668	validation_1-auc:0.941505
[310]	validation_0-auc:0.994756	validation_1-auc:0.941494
[311]	validation_0-auc:0.994757	validation_1-auc:0.9415
[312]	validation_0-auc:0.994776	validation_1-auc:0.941487
[313]	validation_0-auc:0.994818	validation_1-auc:0.941463
[314]	validation_0-auc:0.994868	validation_1-auc:0.941494
[315]	validation_0-auc:0.994922	validation_1-auc:0.941409
[316]	validation_0-auc:0.994963	validation_1-auc:0.941352
[317]	validation_0-auc:0.995003	validation_1-auc:0.941368
[318]	validation_0-auc:0.995001	validation_1-auc:0.941415
[319]	validation_0-auc:0.995052	validation_1-auc:0.941411
[320]	validation_0-auc:0.995067	validation_1-auc:0.941412
[321]	validation_0-auc:0.995102	validation_1-auc:0.941458
[322]	validation_0-auc:0.99512	validation_1-auc:0.941491
[323]	validation_0-auc:0.995151	validation_1-auc:0.941437
[324]	validation_0-auc:0.995181	validation_1-auc:0.941455
[325]	validation_0-auc:0.995245	validation_1-auc:0.941343
[326]	validation_0-auc:0.9953	validation_1-auc:0.941345
[327]	validation_0-auc:0.995314	validation_1-auc:0.941376
[328]	validation_0-auc:0.995328	validation_1-auc:0.941386
[329]	validation_0-auc:0.995392	validation_1-auc:0.941347
[330]	validation_0-auc:0.995427	validation_1-auc:0.941386
[331]	validation_0-auc:0.995476	validation_1-auc:0.941338
[332]	validation_0-auc:0.995531	validation_1-auc:0.941337
[333]	validation_0-auc:0.995579	validation_1-auc:0.941275
[334]	validation_0-auc:0.995631	validation_1-auc:0.941229
[335]	validation_0-auc:0.995659	validation_1-auc:0.941202
[336]	validation_0-auc:0.995687	validation_1-auc:0.941109
[337]	validation_0-auc:0.995725	validation_1-auc:0.941135
[338]	validation_0-auc:0.995736	validation_1-auc:0.941031
[339]	validation_0-auc:0.995762	validation_1-auc:0.940979
[340]	validation_0-auc:0.995806	validation_1-auc:0.940947
[341]	validation_0-auc:0.995832	validation_1-auc:0.940956
[342]	validation_0-auc:0.995864	validation_1-auc:0.940955
[343]	validation_0-auc:0.995889	validation_1-auc:0.940901
[344]	validation_0-auc:0.995923	validation_1-auc:0.940959
[345]	validation_0-auc:0.995956	validation_1-auc:0.94093
[346]	validation_0-auc:0.996011	validation_1-auc:0.940877
[347]	validation_0-auc:0.996009	validation_1-auc:0.940862
[348]	validation_0-auc:0.996033	validation_1-auc:0.940794
[349]	validation_0-auc:0.996075	validation_1-auc:0.94078
[350]	validation_0-auc:0.996104	validation_1-auc:0.940705
[351]	validation_0-auc:0.996128	validation_1-auc:0.940633
[352]	validation_0-auc:0.996193	validation_1-auc:0.94051
[353]	validation_0-auc:0.996225	validation_1-auc:0.940489
[354]	validation_0-auc:0.99623	validation_1-auc:0.940505
[355]	validation_0-auc:0.996257	validation_1-auc:0.94053
[356]	validation_0-auc:0.996288	validation_1-auc:0.94048
[357]	validation_0-auc:0.996315	validation_1-auc:0.94054
[358]	validation_0-auc:0.996355	validation_1-auc:0.940507
[359]	validation_0-auc:0.996372	validation_1-auc:0.940532
[360]	validation_0-auc:0.996409	validation_1-auc:0.940558
[361]	validation_0-auc:0.996431	validation_1-auc:0.940545
[362]	validation_0-auc:0.99649	validation_1-auc:0.940513
[363]	validation_0-auc:0.996505	validation_1-auc:0.940505
[364]	validation_0-auc:0.99653	validation_1-auc:0.940491
[365]	validation_0-auc:0.996558	validation_1-auc:0.940471
[366]	validation_0-auc:0.996573	validation_1-auc:0.94045
[367]	validation_0-auc:0.99663	validation_1-auc:0.940444
[368]	validation_0-auc:0.996639	validation_1-auc:0.940435
[369]	validation_0-auc:0.996641	validation_1-auc:0.940458
[370]	validation_0-auc:0.996679	validation_1-auc:0.940458
[371]	validation_0-auc:0.996721	validation_1-auc:0.94052
[372]	validation_0-auc:0.996738	validation_1-auc:0.940516
[373]	validation_0-auc:0.996766	validation_1-auc:0.940604
[374]	validation_0-auc:0.996787	validation_1-auc:0.940551
[375]	validation_0-auc:0.996805	validation_1-auc:0.940593
[376]	validation_0-auc:0.996837	validation_1-auc:0.940599
[377]	validation_0-auc:0.996868	validation_1-auc:0.940561
[378]	validation_0-auc:0.996878	validation_1-auc:0.940692
[379]	validation_0-auc:0.996898	validation_1-auc:0.940763
[380]	validation_0-auc:0.996908	validation_1-auc:0.940815
[381]	validation_0-auc:0.996903	validation_1-auc:0.940826
[382]	validation_0-auc:0.99692	validation_1-auc:0.940829
[383]	validation_0-auc:0.996955	validation_1-auc:0.940864
[384]	validation_0-auc:0.996982	validation_1-auc:0.940837
[385]	validation_0-auc:0.996995	validation_1-auc:0.940845
[386]	validation_0-auc:0.997021	validation_1-auc:0.940791
[387]	validation_0-auc:0.997031	validation_1-auc:0.940796
[388]	validation_0-auc:0.997069	validation_1-auc:0.940741
[389]	validation_0-auc:0.997101	validation_1-auc:0.94073
[390]	validation_0-auc:0.997142	validation_1-auc:0.940686
[391]	validation_0-auc:0.99716	validation_1-auc:0.940691
[392]	validation_0-auc:0.99718	validation_1-auc:0.940599
[393]	validation_0-auc:0.99719	validation_1-auc:0.940571
[394]	validation_0-auc:0.99724	validation_1-auc:0.940566
[395]	validation_0-auc:0.997284	validation_1-auc:0.940556
[396]	validation_0-auc:0.99728	validation_1-auc:0.940548
[397]	validation_0-auc:0.99729	validation_1-auc:0.940584
[398]	validation_0-auc:0.997304	validation_1-auc:0.940579
[399]	validation_0-auc:0.997328	validation_1-auc:0.940468
[400]	validation_0-auc:0.997353	validation_1-auc:0.940445
[401]	validation_0-auc:0.997394	validation_1-auc:0.940448
[402]	validation_0-auc:0.997412	validation_1-auc:0.940396
[403]	validation_0-auc:0.997413	validation_1-auc:0.940375
[404]	validation_0-auc:0.997432	validation_1-auc:0.940409
[405]	validation_0-auc:0.997452	validation_1-auc:0.940392
[406]	validation_0-auc:0.99747	validation_1-auc:0.940353
[407]	validation_0-auc:0.997506	validation_1-auc:0.940345
[408]	validation_0-auc:0.997533	validation_1-auc:0.940274
[409]	validation_0-auc:0.997551	validation_1-auc:0.94016
[410]	validation_0-auc:0.997582	validation_1-auc:0.940059
[411]	validation_0-auc:0.997596	validation_1-auc:0.939928
[412]	validation_0-auc:0.997601	validation_1-auc:0.939878
[413]	validation_0-auc:0.997598	validation_1-auc:0.93986
[414]	validation_0-auc:0.99761	validation_1-auc:0.939874
[415]	validation_0-auc:0.997616	validation_1-auc:0.939883
[416]	validation_0-auc:0.997615	validation_1-auc:0.939853
[417]	validation_0-auc:0.997644	validation_1-auc:0.939804
[418]	validation_0-auc:0.997664	validation_1-auc:0.939827
[419]	validation_0-auc:0.997675	validation_1-auc:0.939768
[420]	validation_0-auc:0.997685	validation_1-auc:0.939849
[421]	validation_0-auc:0.99773	validation_1-auc:0.939794
[422]	validation_0-auc:0.99776	validation_1-auc:0.939738
[423]	validation_0-auc:0.997772	validation_1-auc:0.939771
[424]	validation_0-auc:0.997789	validation_1-auc:0.939775
[425]	validation_0-auc:0.997786	validation_1-auc:0.939799
[426]	validation_0-auc:0.997789	validation_1-auc:0.939755
[427]	validation_0-auc:0.997793	validation_1-auc:0.939791
[428]	validation_0-auc:0.997831	validation_1-auc:0.939732
[429]	validation_0-auc:0.99784	validation_1-auc:0.939733
[430]	validation_0-auc:0.997859	validation_1-auc:0.939815
[431]	validation_0-auc:0.99787	validation_1-auc:0.939768
[432]	validation_0-auc:0.99788	validation_1-auc:0.939842
[433]	validation_0-auc:0.997904	validation_1-auc:0.939849
[434]	validation_0-auc:0.997916	validation_1-auc:0.939812
[435]	validation_0-auc:0.997911	validation_1-auc:0.939824
[436]	validation_0-auc:0.99792	validation_1-auc:0.93981
[437]	validation_0-auc:0.99793	validation_1-auc:0.939848
[438]	validation_0-auc:0.997925	validation_1-auc:0.93987
[439]	validation_0-auc:0.997943	validation_1-auc:0.939811
[440]	validation_0-auc:0.99796	validation_1-auc:0.939704
[441]	validation_0-auc:0.997958	validation_1-auc:0.939801
[442]	validation_0-auc:0.997994	validation_1-auc:0.939762
[443]	validation_0-auc:0.998009	validation_1-auc:0.93971
[444]	validation_0-auc:0.998033	validation_1-auc:0.939732
[445]	validation_0-auc:0.998044	validation_1-auc:0.939709
[446]	validation_0-auc:0.998053	validation_1-auc:0.939707
[447]	validation_0-auc:0.998062	validation_1-auc:0.939691
[448]	validation_0-auc:0.998075	validation_1-auc:0.939628
[449]	validation_0-auc:0.998094	validation_1-auc:0.939602
[450]	validation_0-auc:0.998109	validation_1-auc:0.93964
[451]	validation_0-auc:0.998116	validation_1-auc:0.93961
[452]	validation_0-auc:0.998135	validation_1-auc:0.93964
[453]	validation_0-auc:0.998152	validation_1-auc:0.939588
[454]	validation_0-auc:0.998154	validation_1-auc:0.939605
[455]	validation_0-auc:0.998157	validation_1-auc:0.939633
[456]	validation_0-auc:0.998159	validation_1-auc:0.939633
[457]	validation_0-auc:0.998171	validation_1-auc:0.939547
[458]	validation_0-auc:0.998169	validation_1-auc:0.93961
[459]	validation_0-auc:0.998189	validation_1-auc:0.939596
[460]	validation_0-auc:0.998226	validation_1-auc:0.939577
[461]	validation_0-auc:0.998243	validation_1-auc:0.939641
[462]	validation_0-auc:0.998253	validation_1-auc:0.939582
[463]	validation_0-auc:0.998272	validation_1-auc:0.939442
[464]	validation_0-auc:0.998282	validation_1-auc:0.939432
[465]	validation_0-auc:0.998292	validation_1-auc:0.939409
[466]	validation_0-auc:0.998286	validation_1-auc:0.939369
[467]	validation_0-auc:0.998293	validation_1-auc:0.939376
[468]	validation_0-auc:0.99831	validation_1-auc:0.939374
[469]	validation_0-auc:0.998321	validation_1-auc:0.939344
[470]	validation_0-auc:0.998337	validation_1-auc:0.93933
[471]	validation_0-auc:0.998341	validation_1-auc:0.939321
[472]	validation_0-auc:0.998364	validation_1-auc:0.939275
[473]	validation_0-auc:0.998368	validation_1-auc:0.9393
[474]	validation_0-auc:0.998372	validation_1-auc:0.939235
[475]	validation_0-auc:0.998375	validation_1-auc:0.939194
[476]	validation_0-auc:0.998381	validation_1-auc:0.939175
[477]	validation_0-auc:0.998399	validation_1-auc:0.939189
[478]	validation_0-auc:0.998405	validation_1-auc:0.939257
[479]	validation_0-auc:0.998412	validation_1-auc:0.939295
[480]	validation_0-auc:0.99842	validation_1-auc:0.939233
[481]	validation_0-auc:0.998431	validation_1-auc:0.939196
[482]	validation_0-auc:0.998445	validation_1-auc:0.939114
[483]	validation_0-auc:0.998457	validation_1-auc:0.939171
[484]	validation_0-auc:0.998476	validation_1-auc:0.939089
[485]	validation_0-auc:0.998507	validation_1-auc:0.939104
[486]	validation_0-auc:0.998506	validation_1-auc:0.939113
[487]	validation_0-auc:0.998506	validation_1-auc:0.939143
[488]	validation_0-auc:0.998508	validation_1-auc:0.939126
[489]	validation_0-auc:0.998529	validation_1-auc:0.93911
[490]	validation_0-auc:0.998556	validation_1-auc:0.939112
[491]	validation_0-auc:0.998567	validation_1-auc:0.93917
[492]	validation_0-auc:0.998578	validation_1-auc:0.939144
[493]	validation_0-auc:0.998578	validation_1-auc:0.939209
[494]	validation_0-auc:0.998594	validation_1-auc:0.939212
[495]	validation_0-auc:0.998606	validation_1-auc:0.939195
[496]	validation_0-auc:0.998625	validation_1-auc:0.939137
[497]	validation_0-auc:0.998648	validation_1-auc:0.939128
[498]	validation_0-auc:0.998645	validation_1-auc:0.939139
[499]	validation_0-auc:0.998657	validation_1-auc:0.939166
[500]	validation_0-auc:0.99867	validation_1-auc:0.939143
[501]	validation_0-auc:0.998677	validation_1-auc:0.939155
[502]	validation_0-auc:0.998689	validation_1-auc:0.939157
[503]	validation_0-auc:0.998701	validation_1-auc:0.939156
[504]	validation_0-auc:0.998724	validation_1-auc:0.939169
[505]	validation_0-auc:0.998734	validation_1-auc:0.939178
[506]	validation_0-auc:0.998742	validation_1-auc:0.939127
[507]	validation_0-auc:0.998754	validation_1-auc:0.939123
[508]	validation_0-auc:0.998754	validation_1-auc:0.939165
[509]	validation_0-auc:0.998764	validation_1-auc:0.939138
[510]	validation_0-auc:0.99877	validation_1-auc:0.939161
[511]	validation_0-auc:0.998773	validation_1-auc:0.939168
[512]	validation_0-auc:0.998781	validation_1-auc:0.939173
[513]	validation_0-auc:0.998787	validation_1-auc:0.939183
[514]	validation_0-auc:0.998805	validation_1-auc:0.93909
[515]	validation_0-auc:0.998831	validation_1-auc:0.939127
[516]	validation_0-auc:0.998841	validation_1-auc:0.939111
[517]	validation_0-auc:0.998851	validation_1-auc:0.939088
[518]	validation_0-auc:0.998862	validation_1-auc:0.9391
[519]	validation_0-auc:0.998874	validation_1-auc:0.93907
[520]	validation_0-auc:0.998877	validation_1-auc:0.939086
[521]	validation_0-auc:0.998884	validation_1-auc:0.939072
[522]	validation_0-auc:0.998895	validation_1-auc:0.939098
[523]	validation_0-auc:0.998901	validation_1-auc:0.939049
[524]	validation_0-auc:0.998907	validation_1-auc:0.939033
[525]	validation_0-auc:0.998908	validation_1-auc:0.939085
[526]	validation_0-auc:0.998915	validation_1-auc:0.939073
[527]	validation_0-auc:0.998922	validation_1-auc:0.939162
[528]	validation_0-auc:0.998926	validation_1-auc:0.939141
[529]	validation_0-auc:0.998948	validation_1-auc:0.939126
[530]	validation_0-auc:0.998963	validation_1-auc:0.939091
[531]	validation_0-auc:0.998963	validation_1-auc:0.939089
[532]	validation_0-auc:0.998971	validation_1-auc:0.939103
[533]	validation_0-auc:0.998972	validation_1-auc:0.93914
[534]	validation_0-auc:0.998976	validation_1-auc:0.939159
[535]	validation_0-auc:0.998979	validation_1-auc:0.939252
[536]	validation_0-auc:0.99898	validation_1-auc:0.939308
[537]	validation_0-auc:0.998988	validation_1-auc:0.939273
[538]	validation_0-auc:0.998995	validation_1-auc:0.939406
[539]	validation_0-auc:0.999005	validation_1-auc:0.939392
[540]	validation_0-auc:0.999011	validation_1-auc:0.939342
[541]	validation_0-auc:0.999018	validation_1-auc:0.939416
[542]	validation_0-auc:0.99902	validation_1-auc:0.939379
[543]	validation_0-auc:0.999027	validation_1-auc:0.939416
[544]	validation_0-auc:0.999029	validation_1-auc:0.939352
[545]	validation_0-auc:0.999038	validation_1-auc:0.939399
[546]	validation_0-auc:0.999048	validation_1-auc:0.939334
[547]	validation_0-auc:0.999063	validation_1-auc:0.939326
[548]	validation_0-auc:0.999079	validation_1-auc:0.939378
[549]	validation_0-auc:0.999089	validation_1-auc:0.939346
[550]	validation_0-auc:0.999101	validation_1-auc:0.939275
[551]	validation_0-auc:0.999107	validation_1-auc:0.939253
[552]	validation_0-auc:0.999113	validation_1-auc:0.939289
[553]	validation_0-auc:0.999112	validation_1-auc:0.939296
[554]	validation_0-auc:0.999117	validation_1-auc:0.939301
[555]	validation_0-auc:0.999122	validation_1-auc:0.939238
[556]	validation_0-auc:0.999128	validation_1-auc:0.939296
[557]	validation_0-auc:0.999138	validation_1-auc:0.939268
[558]	validation_0-auc:0.999141	validation_1-auc:0.939336
[559]	validation_0-auc:0.999151	validation_1-auc:0.939303
[560]	validation_0-auc:0.999161	validation_1-auc:0.939205
[561]	validation_0-auc:0.999166	validation_1-auc:0.939193
[562]	validation_0-auc:0.999168	validation_1-auc:0.93921
[563]	validation_0-auc:0.999178	validation_1-auc:0.939248
[564]	validation_0-auc:0.999185	validation_1-auc:0.939276
[565]	validation_0-auc:0.999198	validation_1-auc:0.939139
[566]	validation_0-auc:0.999209	validation_1-auc:0.939166
[567]	validation_0-auc:0.99922	validation_1-auc:0.939097
[568]	validation_0-auc:0.999229	validation_1-auc:0.939079
[569]	validation_0-auc:0.999242	validation_1-auc:0.939062
[570]	validation_0-auc:0.999247	validation_1-auc:0.938993
[571]	validation_0-auc:0.99925	validation_1-auc:0.939049
[572]	validation_0-auc:0.999259	validation_1-auc:0.939071
[573]	validation_0-auc:0.999263	validation_1-auc:0.939028
[574]	validation_0-auc:0.999265	validation_1-auc:0.938972
[575]	validation_0-auc:0.999269	validation_1-auc:0.93894
[576]	validation_0-auc:0.999266	validation_1-auc:0.938935
[577]	validation_0-auc:0.999273	validation_1-auc:0.938932
[578]	validation_0-auc:0.999283	validation_1-auc:0.938916
[579]	validation_0-auc:0.999291	validation_1-auc:0.938797
[580]	validation_0-auc:0.999296	validation_1-auc:0.938828
[581]	validation_0-auc:0.999296	validation_1-auc:0.938821
[582]	validation_0-auc:0.999301	validation_1-auc:0.93886
[583]	validation_0-auc:0.999308	validation_1-auc:0.938783
[584]	validation_0-auc:0.99931	validation_1-auc:0.938713
[585]	validation_0-auc:0.999322	validation_1-auc:0.938764
[586]	validation_0-auc:0.999335	validation_1-auc:0.938765
[587]	validation_0-auc:0.999339	validation_1-auc:0.938776
[588]	validation_0-auc:0.99934	validation_1-auc:0.938734
[589]	validation_0-auc:0.999345	validation_1-auc:0.938735
[590]	validation_0-auc:0.999352	validation_1-auc:0.93878
[591]	validation_0-auc:0.999357	validation_1-auc:0.938807
[592]	validation_0-auc:0.999358	validation_1-auc:0.938866
[593]	validation_0-auc:0.999362	validation_1-auc:0.938847
[594]	validation_0-auc:0.999371	validation_1-auc:0.938815
[595]	validation_0-auc:0.999375	validation_1-auc:0.938888
[596]	validation_0-auc:0.999392	validation_1-auc:0.938894
[597]	validation_0-auc:0.9994	validation_1-auc:0.938894
[598]	validation_0-auc:0.999401	validation_1-auc:0.938894
[599]	validation_0-auc:0.999397	validation_1-auc:0.938951
[600]	validation_0-auc:0.999398	validation_1-auc:0.938962
[601]	validation_0-auc:0.99941	validation_1-auc:0.938962
[602]	validation_0-auc:0.999418	validation_1-auc:0.938971
[603]	validation_0-auc:0.999414	validation_1-auc:0.938981
[604]	validation_0-auc:0.999421	validation_1-auc:0.938991
[605]	validation_0-auc:0.999427	validation_1-auc:0.938955
[606]	validation_0-auc:0.999435	validation_1-auc:0.938932
[607]	validation_0-auc:0.999444	validation_1-auc:0.93898
[608]	validation_0-auc:0.999448	validation_1-auc:0.938971
[609]	validation_0-auc:0.999455	validation_1-auc:0.938892
[610]	validation_0-auc:0.999467	validation_1-auc:0.938887
[611]	validation_0-auc:0.999471	validation_1-auc:0.938876
[612]	validation_0-auc:0.99947	validation_1-auc:0.938867
[613]	validation_0-auc:0.99947	validation_1-auc:0.938847
[614]	validation_0-auc:0.999468	validation_1-auc:0.938827
[615]	validation_0-auc:0.999469	validation_1-auc:0.938892
[616]	validation_0-auc:0.999475	validation_1-auc:0.938924
[617]	validation_0-auc:0.999473	validation_1-auc:0.938886
[618]	validation_0-auc:0.999476	validation_1-auc:0.93887
[619]	validation_0-auc:0.999479	validation_1-auc:0.938818
[620]	validation_0-auc:0.999484	validation_1-auc:0.938792
[621]	validation_0-auc:0.999481	validation_1-auc:0.938765
[622]	validation_0-auc:0.999489	validation_1-auc:0.938827
[623]	validation_0-auc:0.999494	validation_1-auc:0.938801
[624]	validation_0-auc:0.999491	validation_1-auc:0.938846
[625]	validation_0-auc:0.999497	validation_1-auc:0.938856
[626]	validation_0-auc:0.999495	validation_1-auc:0.938945
[627]	validation_0-auc:0.999502	validation_1-auc:0.938859
[628]	validation_0-auc:0.999501	validation_1-auc:0.93888
[629]	validation_0-auc:0.999504	validation_1-auc:0.938905
[630]	validation_0-auc:0.999503	validation_1-auc:0.938917
[631]	validation_0-auc:0.999508	validation_1-auc:0.938913
[632]	validation_0-auc:0.999508	validation_1-auc:0.938888
[633]	validation_0-auc:0.999516	validation_1-auc:0.938919
[634]	validation_0-auc:0.999518	validation_1-auc:0.938915
[635]	validation_0-auc:0.999521	validation_1-auc:0.938908
[636]	validation_0-auc:0.999523	validation_1-auc:0.938947
[637]	validation_0-auc:0.999528	validation_1-auc:0.938917
[638]	validation_0-auc:0.999536	validation_1-auc:0.938921
[639]	validation_0-auc:0.999548	validation_1-auc:0.939001
[640]	validation_0-auc:0.999552	validation_1-auc:0.93899
[641]	validation_0-auc:0.999556	validation_1-auc:0.938993
[642]	validation_0-auc:0.999561	validation_1-auc:0.938972
[643]	validation_0-auc:0.999559	validation_1-auc:0.938975
[644]	validation_0-auc:0.99956	validation_1-auc:0.938944
[645]	validation_0-auc:0.999571	validation_1-auc:0.938916
[646]	validation_0-auc:0.999575	validation_1-auc:0.938915
[647]	validation_0-auc:0.999577	validation_1-auc:0.938819
[648]	validation_0-auc:0.999581	validation_1-auc:0.938834
[649]	validation_0-auc:0.999584	validation_1-auc:0.938839
[650]	validation_0-auc:0.999585	validation_1-auc:0.938809
[651]	validation_0-auc:0.99959	validation_1-auc:0.938839
[652]	validation_0-auc:0.999589	validation_1-auc:0.938841
[653]	validation_0-auc:0.999589	validation_1-auc:0.938841
[654]	validation_0-auc:0.99959	validation_1-auc:0.938864
[655]	validation_0-auc:0.999594	validation_1-auc:0.938878
[656]	validation_0-auc:0.999593	validation_1-auc:0.938848
[657]	validation_0-auc:0.999594	validation_1-auc:0.938827
[658]	validation_0-auc:0.999596	validation_1-auc:0.938815
[659]	validation_0-auc:0.999605	validation_1-auc:0.938782
[660]	validation_0-auc:0.999608	validation_1-auc:0.938766
[661]	validation_0-auc:0.999613	validation_1-auc:0.938712
[662]	validation_0-auc:0.999623	validation_1-auc:0.938685
[663]	validation_0-auc:0.999625	validation_1-auc:0.938683
[664]	validation_0-auc:0.999626	validation_1-auc:0.938617
[665]	validation_0-auc:0.999626	validation_1-auc:0.938585
[666]	validation_0-auc:0.999625	validation_1-auc:0.938602
[667]	validation_0-auc:0.999629	validation_1-auc:0.938607
[668]	validation_0-auc:0.999636	validation_1-auc:0.938545
[669]	validation_0-auc:0.999635	validation_1-auc:0.938504
[670]	validation_0-auc:0.999643	validation_1-auc:0.938545
[671]	validation_0-auc:0.999645	validation_1-auc:0.938518
[672]	validation_0-auc:0.99965	validation_1-auc:0.938593
[673]	validation_0-auc:0.999652	validation_1-auc:0.938584
[674]	validation_0-auc:0.999653	validation_1-auc:0.938549
[675]	validation_0-auc:0.999659	validation_1-auc:0.938561
[676]	validation_0-auc:0.999662	validation_1-auc:0.938596
[677]	validation_0-auc:0.999664	validation_1-auc:0.938586
[678]	validation_0-auc:0.999666	validation_1-auc:0.938568
[679]	validation_0-auc:0.99967	validation_1-auc:0.938616
[680]	validation_0-auc:0.999675	validation_1-auc:0.938527
[681]	validation_0-auc:0.999676	validation_1-auc:0.938569
[682]	validation_0-auc:0.999682	validation_1-auc:0.938584
[683]	validation_0-auc:0.999683	validation_1-auc:0.938568
[684]	validation_0-auc:0.999687	validation_1-auc:0.938538
[685]	validation_0-auc:0.999689	validation_1-auc:0.938456
[686]	validation_0-auc:0.99969	validation_1-auc:0.938477
[687]	validation_0-auc:0.99969	validation_1-auc:0.938476
[688]	validation_0-auc:0.999694	validation_1-auc:0.938446
[689]	validation_0-auc:0.999699	validation_1-auc:0.938479
[690]	validation_0-auc:0.999702	validation_1-auc:0.93848
[691]	validation_0-auc:0.999703	validation_1-auc:0.938394
[692]	validation_0-auc:0.999707	validation_1-auc:0.938408
[693]	validation_0-auc:0.999711	validation_1-auc:0.938428
[694]	validation_0-auc:0.999711	validation_1-auc:0.938384
[695]	validation_0-auc:0.999713	validation_1-auc:0.938389
[696]	validation_0-auc:0.999716	validation_1-auc:0.9384
[697]	validation_0-auc:0.999717	validation_1-auc:0.938372
[698]	validation_0-auc:0.999727	validation_1-auc:0.938293
[699]	validation_0-auc:0.999728	validation_1-auc:0.938304
[700]	validation_0-auc:0.999732	validation_1-auc:0.938322
[701]	validation_0-auc:0.99974	validation_1-auc:0.938338
[702]	validation_0-auc:0.999742	validation_1-auc:0.9383
[703]	validation_0-auc:0.999741	validation_1-auc:0.938324
[704]	validation_0-auc:0.999745	validation_1-auc:0.938377
[705]	validation_0-auc:0.99975	validation_1-auc:0.938384
[706]	validation_0-auc:0.999751	validation_1-auc:0.938402
[707]	validation_0-auc:0.999751	validation_1-auc:0.938435
[708]	validation_0-auc:0.999751	validation_1-auc:0.938474
[709]	validation_0-auc:0.999753	validation_1-auc:0.938505
[710]	validation_0-auc:0.999755	validation_1-auc:0.938471
[711]	validation_0-auc:0.999757	validation_1-auc:0.938457
[712]	validation_0-auc:0.999756	validation_1-auc:0.938475
[713]	validation_0-auc:0.999763	validation_1-auc:0.938447
[714]	validation_0-auc:0.999769	validation_1-auc:0.938423
[715]	validation_0-auc:0.999772	validation_1-auc:0.938389
[716]	validation_0-auc:0.999774	validation_1-auc:0.938361
[717]	validation_0-auc:0.99978	validation_1-auc:0.938336
[718]	validation_0-auc:0.999781	validation_1-auc:0.938294
[719]	validation_0-auc:0.999781	validation_1-auc:0.938274
[720]	validation_0-auc:0.999778	validation_1-auc:0.93826
[721]	validation_0-auc:0.999777	validation_1-auc:0.938227
[722]	validation_0-auc:0.999783	validation_1-auc:0.938164
[723]	validation_0-auc:0.999787	validation_1-auc:0.938241
[724]	validation_0-auc:0.999787	validation_1-auc:0.938229
[725]	validation_0-auc:0.999788	validation_1-auc:0.938166
[726]	validation_0-auc:0.999789	validation_1-auc:0.938109
[727]	validation_0-auc:0.999791	validation_1-auc:0.938028
[728]	validation_0-auc:0.999792	validation_1-auc:0.938005
[729]	validation_0-auc:0.999794	validation_1-auc:0.93798
[730]	validation_0-auc:0.999791	validation_1-auc:0.938024
[731]	validation_0-auc:0.999793	validation_1-auc:0.938068
[732]	validation_0-auc:0.999792	validation_1-auc:0.938099
[733]	validation_0-auc:0.999797	validation_1-auc:0.938044
[734]	validation_0-auc:0.999797	validation_1-auc:0.938098
[735]	validation_0-auc:0.9998	validation_1-auc:0.938056
[736]	validation_0-auc:0.999802	validation_1-auc:0.93806
[737]	validation_0-auc:0.999806	validation_1-auc:0.93802
[738]	validation_0-auc:0.999806	validation_1-auc:0.938013
[739]	validation_0-auc:0.999807	validation_1-auc:0.938051
[740]	validation_0-auc:0.999807	validation_1-auc:0.938002
[741]	validation_0-auc:0.99981	validation_1-auc:0.937952
[742]	validation_0-auc:0.999811	validation_1-auc:0.937946
[743]	validation_0-auc:0.999811	validation_1-auc:0.937973
[744]	validation_0-auc:0.999818	validation_1-auc:0.937905
[745]	validation_0-auc:0.999819	validation_1-auc:0.937867
[746]	validation_0-auc:0.999825	validation_1-auc:0.937903
[747]	validation_0-auc:0.999826	validation_1-auc:0.937893
[748]	validation_0-auc:0.999828	validation_1-auc:0.93782
[749]	validation_0-auc:0.999827	validation_1-auc:0.937749
[750]	validation_0-auc:0.999828	validation_1-auc:0.937768
[751]	validation_0-auc:0.999831	validation_1-auc:0.937744
[752]	validation_0-auc:0.999831	validation_1-auc:0.937721
[753]	validation_0-auc:0.999835	validation_1-auc:0.937704
[754]	validation_0-auc:0.999833	validation_1-auc:0.937687
[755]	validation_0-auc:0.999838	validation_1-auc:0.937715
[756]	validation_0-auc:0.999839	validation_1-auc:0.937739
[757]	validation_0-auc:0.99984	validation_1-auc:0.937784
[758]	validation_0-auc:0.999841	validation_1-auc:0.937835
[759]	validation_0-auc:0.999844	validation_1-auc:0.937855
[760]	validation_0-auc:0.999847	validation_1-auc:0.937862
[761]	validation_0-auc:0.999846	validation_1-auc:0.93791
[762]	validation_0-auc:0.999847	validation_1-auc:0.937875
[763]	validation_0-auc:0.999849	validation_1-auc:0.937898
[764]	validation_0-auc:0.999851	validation_1-auc:0.937986
[765]	validation_0-auc:0.999849	validation_1-auc:0.937927
[766]	validation_0-auc:0.99985	validation_1-auc:0.937912
[767]	validation_0-auc:0.999857	validation_1-auc:0.937829
[768]	validation_0-auc:0.999858	validation_1-auc:0.937802
[769]	validation_0-auc:0.999861	validation_1-auc:0.937859
[770]	validation_0-auc:0.999866	validation_1-auc:0.937829
[771]	validation_0-auc:0.999867	validation_1-auc:0.937846
[772]	validation_0-auc:0.999869	validation_1-auc:0.937784
[773]	validation_0-auc:0.999869	validation_1-auc:0.937785
[774]	validation_0-auc:0.999875	validation_1-auc:0.937804
[775]	validation_0-auc:0.999875	validation_1-auc:0.93783
[776]	validation_0-auc:0.999877	validation_1-auc:0.937743
[777]	validation_0-auc:0.999878	validation_1-auc:0.937734
[778]	validation_0-auc:0.999881	validation_1-auc:0.937762
[779]	validation_0-auc:0.999881	validation_1-auc:0.937818
[780]	validation_0-auc:0.999885	validation_1-auc:0.93779
[781]	validation_0-auc:0.999886	validation_1-auc:0.937775
[782]	validation_0-auc:0.999887	validation_1-auc:0.937784
[783]	validation_0-auc:0.999889	validation_1-auc:0.937799
[784]	validation_0-auc:0.99989	validation_1-auc:0.937745
[785]	validation_0-auc:0.99989	validation_1-auc:0.937737
[786]	validation_0-auc:0.999889	validation_1-auc:0.937755
[787]	validation_0-auc:0.999892	validation_1-auc:0.937724
[788]	validation_0-auc:0.999893	validation_1-auc:0.937649
[789]	validation_0-auc:0.999893	validation_1-auc:0.937657
[790]	validation_0-auc:0.999894	validation_1-auc:0.937663
[791]	validation_0-auc:0.999894	validation_1-auc:0.937619
[792]	validation_0-auc:0.999893	validation_1-auc:0.93767
[793]	validation_0-auc:0.999896	validation_1-auc:0.937716
[794]	validation_0-auc:0.999894	validation_1-auc:0.937764
[795]	validation_0-auc:0.999896	validation_1-auc:0.937773
[796]	validation_0-auc:0.999899	validation_1-auc:0.937708
[797]	validation_0-auc:0.999901	validation_1-auc:0.937745
[798]	validation_0-auc:0.999901	validation_1-auc:0.937738
[799]	validation_0-auc:0.999902	validation_1-auc:0.937759
[800]	validation_0-auc:0.999902	validation_1-auc:0.937767
[801]	validation_0-auc:0.999904	validation_1-auc:0.937808
[802]	validation_0-auc:0.999907	validation_1-auc:0.937779
[803]	validation_0-auc:0.999908	validation_1-auc:0.937788
[804]	validation_0-auc:0.999909	validation_1-auc:0.937853
[805]	validation_0-auc:0.999911	validation_1-auc:0.937823
[806]	validation_0-auc:0.99991	validation_1-auc:0.937818
[807]	validation_0-auc:0.99991	validation_1-auc:0.937785
[808]	validation_0-auc:0.999912	validation_1-auc:0.937767
[809]	validation_0-auc:0.99991	validation_1-auc:0.937776
[810]	validation_0-auc:0.999912	validation_1-auc:0.937791
[811]	validation_0-auc:0.999915	validation_1-auc:0.937729
[812]	validation_0-auc:0.999915	validation_1-auc:0.93773
[813]	validation_0-auc:0.999915	validation_1-auc:0.93773
[814]	validation_0-auc:0.999915	validation_1-auc:0.937721
[815]	validation_0-auc:0.999918	validation_1-auc:0.937695
[816]	validation_0-auc:0.99992	validation_1-auc:0.937713
[817]	validation_0-auc:0.99992	validation_1-auc:0.937693
[818]	validation_0-auc:0.99992	validation_1-auc:0.937653
[819]	validation_0-auc:0.999923	validation_1-auc:0.937661
[820]	validation_0-auc:0.999924	validation_1-auc:0.937661
[821]	validation_0-auc:0.999926	validation_1-auc:0.937661
[822]	validation_0-auc:0.999925	validation_1-auc:0.937646
[823]	validation_0-auc:0.999925	validation_1-auc:0.937596
[824]	validation_0-auc:0.999925	validation_1-auc:0.937555
[825]	validation_0-auc:0.999925	validation_1-auc:0.937591
[826]	validation_0-auc:0.999926	validation_1-auc:0.937555
[827]	validation_0-auc:0.999926	validation_1-auc:0.937554
[828]	validation_0-auc:0.999927	validation_1-auc:0.937525
[829]	validation_0-auc:0.999927	validation_1-auc:0.937526
[830]	validation_0-auc:0.999927	validation_1-auc:0.937526
[831]	validation_0-auc:0.999929	validation_1-auc:0.937584
[832]	validation_0-auc:0.999932	validation_1-auc:0.937609
[833]	validation_0-auc:0.999933	validation_1-auc:0.937629
[834]	validation_0-auc:0.999934	validation_1-auc:0.937672
[835]	validation_0-auc:0.999935	validation_1-auc:0.937691
[836]	validation_0-auc:0.999939	validation_1-auc:0.937708
[837]	validation_0-auc:0.999937	validation_1-auc:0.937641
[838]	validation_0-auc:0.999938	validation_1-auc:0.937586
[839]	validation_0-auc:0.999937	validation_1-auc:0.93754
[840]	validation_0-auc:0.999937	validation_1-auc:0.937513
[841]	validation_0-auc:0.999938	validation_1-auc:0.937462
[842]	validation_0-auc:0.999939	validation_1-auc:0.937383
[843]	validation_0-auc:0.999938	validation_1-auc:0.937336
[844]	validation_0-auc:0.999938	validation_1-auc:0.937334
[845]	validation_0-auc:0.999938	validation_1-auc:0.937373
[846]	validation_0-auc:0.99994	validation_1-auc:0.937337
[847]	validation_0-auc:0.99994	validation_1-auc:0.937334
[848]	validation_0-auc:0.999941	validation_1-auc:0.937304
[849]	validation_0-auc:0.99994	validation_1-auc:0.937306
[850]	validation_0-auc:0.999941	validation_1-auc:0.937332
[851]	validation_0-auc:0.999941	validation_1-auc:0.937336
[852]	validation_0-auc:0.999943	validation_1-auc:0.937312
[853]	validation_0-auc:0.999945	validation_1-auc:0.937361
[854]	validation_0-auc:0.999944	validation_1-auc:0.937346
[855]	validation_0-auc:0.999944	validation_1-auc:0.937316
[856]	validation_0-auc:0.999944	validation_1-auc:0.937294
[857]	validation_0-auc:0.999944	validation_1-auc:0.937233
[858]	validation_0-auc:0.999945	validation_1-auc:0.93723
[859]	validation_0-auc:0.999947	validation_1-auc:0.937262
[860]	validation_0-auc:0.999948	validation_1-auc:0.937243
[861]	validation_0-auc:0.999949	validation_1-auc:0.937226
[862]	validation_0-auc:0.99995	validation_1-auc:0.937275
[863]	validation_0-auc:0.999951	validation_1-auc:0.937316
[864]	validation_0-auc:0.999951	validation_1-auc:0.93735
[865]	validation_0-auc:0.999951	validation_1-auc:0.937325
[866]	validation_0-auc:0.99995	validation_1-auc:0.937352
[867]	validation_0-auc:0.99995	validation_1-auc:0.937331
[868]	validation_0-auc:0.999951	validation_1-auc:0.937318
[869]	validation_0-auc:0.999952	validation_1-auc:0.937359
[870]	validation_0-auc:0.999954	validation_1-auc:0.937328
[871]	validation_0-auc:0.999954	validation_1-auc:0.937326
[872]	validation_0-auc:0.999953	validation_1-auc:0.937409
[873]	validation_0-auc:0.999952	validation_1-auc:0.937403
[874]	validation_0-auc:0.999952	validation_1-auc:0.937362
[875]	validation_0-auc:0.999954	validation_1-auc:0.937333
[876]	validation_0-auc:0.999956	validation_1-auc:0.937355
[877]	validation_0-auc:0.999954	validation_1-auc:0.93735
[878]	validation_0-auc:0.999956	validation_1-auc:0.937344
[879]	validation_0-auc:0.999957	validation_1-auc:0.937308
[880]	validation_0-auc:0.999958	validation_1-auc:0.937337
[881]	validation_0-auc:0.999959	validation_1-auc:0.93729
[882]	validation_0-auc:0.999959	validation_1-auc:0.937282
[883]	validation_0-auc:0.999959	validation_1-auc:0.937233
[884]	validation_0-auc:0.999959	validation_1-auc:0.937231
[885]	validation_0-auc:0.999961	validation_1-auc:0.937237
[886]	validation_0-auc:0.999963	validation_1-auc:0.937199
[887]	validation_0-auc:0.999965	validation_1-auc:0.937281
[888]	validation_0-auc:0.999964	validation_1-auc:0.937247
[889]	validation_0-auc:0.999964	validation_1-auc:0.937283
[890]	validation_0-auc:0.999964	validation_1-auc:0.937237
[891]	validation_0-auc:0.999964	validation_1-auc:0.937249
[892]	validation_0-auc:0.999963	validation_1-auc:0.937214
[893]	validation_0-auc:0.999964	validation_1-auc:0.937257
[894]	validation_0-auc:0.999966	validation_1-auc:0.937302
[895]	validation_0-auc:0.999967	validation_1-auc:0.937289
[896]	validation_0-auc:0.999966	validation_1-auc:0.937359
[897]	validation_0-auc:0.999968	validation_1-auc:0.93734
[898]	validation_0-auc:0.999969	validation_1-auc:0.937389
[899]	validation_0-auc:0.999969	validation_1-auc:0.937396
[900]	validation_0-auc:0.999969	validation_1-auc:0.937425
[901]	validation_0-auc:0.999971	validation_1-auc:0.937475
[902]	validation_0-auc:0.999971	validation_1-auc:0.937419
[903]	validation_0-auc:0.999971	validation_1-auc:0.937417
[904]	validation_0-auc:0.999971	validation_1-auc:0.937425
[905]	validation_0-auc:0.999972	validation_1-auc:0.937436
[906]	validation_0-auc:0.999972	validation_1-auc:0.937357
[907]	validation_0-auc:0.999972	validation_1-auc:0.937375
[908]	validation_0-auc:0.999974	validation_1-auc:0.937317
[909]	validation_0-auc:0.999974	validation_1-auc:0.937404
[910]	validation_0-auc:0.999975	validation_1-auc:0.937387
[911]	validation_0-auc:0.999974	validation_1-auc:0.93741
[912]	validation_0-auc:0.999974	validation_1-auc:0.937482
[913]	validation_0-auc:0.999974	validation_1-auc:0.937433
[914]	validation_0-auc:0.999973	validation_1-auc:0.93745
[915]	validation_0-auc:0.999973	validation_1-auc:0.937483
[916]	validation_0-auc:0.999973	validation_1-auc:0.937436
[917]	validation_0-auc:0.999973	validation_1-auc:0.93741
[918]	validation_0-auc:0.999974	validation_1-auc:0.937435
[919]	validation_0-auc:0.999973	validation_1-auc:0.937401
[920]	validation_0-auc:0.999974	validation_1-auc:0.937434
[921]	validation_0-auc:0.999974	validation_1-auc:0.937447
[922]	validation_0-auc:0.999975	validation_1-auc:0.937509
[923]	validation_0-auc:0.999976	validation_1-auc:0.937517
[924]	validation_0-auc:0.999976	validation_1-auc:0.937488
[925]	validation_0-auc:0.999976	validation_1-auc:0.937538
[926]	validation_0-auc:0.999977	validation_1-auc:0.937519
[927]	validation_0-auc:0.999977	validation_1-auc:0.937513
[928]	validation_0-auc:0.999977	validation_1-auc:0.937534
[929]	validation_0-auc:0.999977	validation_1-auc:0.93756
[930]	validation_0-auc:0.999978	validation_1-auc:0.93755
[931]	validation_0-auc:0.999977	validation_1-auc:0.937512
[932]	validation_0-auc:0.999979	validation_1-auc:0.937533
[933]	validation_0-auc:0.999979	validation_1-auc:0.937493
[934]	validation_0-auc:0.99998	validation_1-auc:0.9375
[935]	validation_0-auc:0.999981	validation_1-auc:0.937452
[936]	validation_0-auc:0.999981	validation_1-auc:0.937451
[937]	validation_0-auc:0.999981	validation_1-auc:0.937415
[938]	validation_0-auc:0.999981	validation_1-auc:0.937428
[939]	validation_0-auc:0.999981	validation_1-auc:0.937397
[940]	validation_0-auc:0.999981	validation_1-auc:0.937401
[941]	validation_0-auc:0.99998	validation_1-auc:0.937397
[942]	validation_0-auc:0.999981	validation_1-auc:0.937424
[943]	validation_0-auc:0.999981	validation_1-auc:0.937432
[944]	validation_0-auc:0.99998	validation_1-auc:0.937385
[945]	validation_0-auc:0.999979	validation_1-auc:0.937374
[946]	validation_0-auc:0.99998	validation_1-auc:0.937358
[947]	validation_0-auc:0.99998	validation_1-auc:0.937381
[948]	validation_0-auc:0.999981	validation_1-auc:0.937413
[949]	validation_0-auc:0.999981	validation_1-auc:0.937374
[950]	validation_0-auc:0.999981	validation_1-auc:0.93741
[951]	validation_0-auc:0.999981	validation_1-auc:0.937363
[952]	validation_0-auc:0.999981	validation_1-auc:0.937342
[953]	validation_0-auc:0.999981	validation_1-auc:0.937265
[954]	validation_0-auc:0.99998	validation_1-auc:0.937199
[955]	validation_0-auc:0.99998	validation_1-auc:0.937177
[956]	validation_0-auc:0.99998	validation_1-auc:0.937137
[957]	validation_0-auc:0.99998	validation_1-auc:0.937109
[958]	validation_0-auc:0.999981	validation_1-auc:0.937082
[959]	validation_0-auc:0.999981	validation_1-auc:0.937045
[960]	validation_0-auc:0.999981	validation_1-auc:0.937068
[961]	validation_0-auc:0.999981	validation_1-auc:0.937052
[962]	validation_0-auc:0.999982	validation_1-auc:0.937015
[963]	validation_0-auc:0.999982	validation_1-auc:0.936991
[964]	validation_0-auc:0.999982	validation_1-auc:0.937057
[965]	validation_0-auc:0.999982	validation_1-auc:0.937083
[966]	validation_0-auc:0.999982	validation_1-auc:0.937094
[967]	validation_0-auc:0.999983	validation_1-auc:0.937109
[968]	validation_0-auc:0.999982	validation_1-auc:0.937117
[969]	validation_0-auc:0.999983	validation_1-auc:0.937132
[970]	validation_0-auc:0.999983	validation_1-auc:0.937109
[971]	validation_0-auc:0.999983	validation_1-auc:0.937076
[972]	validation_0-auc:0.999985	validation_1-auc:0.937085
[973]	validation_0-auc:0.999984	validation_1-auc:0.937049
[974]	validation_0-auc:0.999985	validation_1-auc:0.937048
[975]	validation_0-auc:0.999985	validation_1-auc:0.937025
[976]	validation_0-auc:0.999985	validation_1-auc:0.937039
[977]	validation_0-auc:0.999985	validation_1-auc:0.937067
[978]	validation_0-auc:0.999986	validation_1-auc:0.937081
[979]	validation_0-auc:0.999985	validation_1-auc:0.93706
[980]	validation_0-auc:0.999986	validation_1-auc:0.937074
[981]	validation_0-auc:0.999987	validation_1-auc:0.937108
[982]	validation_0-auc:0.999987	validation_1-auc:0.937112
[983]	validation_0-auc:0.999987	validation_1-auc:0.937093
[984]	validation_0-auc:0.999986	validation_1-auc:0.937109
[985]	validation_0-auc:0.999986	validation_1-auc:0.937027
[986]	validation_0-auc:0.999986	validation_1-auc:0.937016
[987]	validation_0-auc:0.999986	validation_1-auc:0.937023
[988]	validation_0-auc:0.999986	validation_1-auc:0.93702
[989]	validation_0-auc:0.999986	validation_1-auc:0.936998
[990]	validation_0-auc:0.999986	validation_1-auc:0.937026
[991]	validation_0-auc:0.999986	validation_1-auc:0.936997
[992]	validation_0-auc:0.999986	validation_1-auc:0.937019
[993]	validation_0-auc:0.999986	validation_1-auc:0.937035
[994]	validation_0-auc:0.999985	validation_1-auc:0.936997
[995]	validation_0-auc:0.999984	validation_1-auc:0.937048
[996]	validation_0-auc:0.999985	validation_1-auc:0.937058
[997]	validation_0-auc:0.999985	validation_1-auc:0.937005
[998]	validation_0-auc:0.999985	validation_1-auc:0.937091
[999]	validation_0-auc:0.999984	validation_1-auc:0.937047
Wall time: 29.2 s
Out[102]:
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1,
              disable_default_eval_metric=1, eval_metric='auc', gamma=0.15,
              learning_rate=0.1, max_delta_step=0, max_depth=7,
              min_child_weight=3, missing=None, n_estimators=1000, n_jobs=8,
              nthread=None, objective='binary:logistic', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=7.8243170862346005,
              seed=None, silent=None, subsample=0.5, verbose_eval=100,
              verbosity=0)
In [103]:
true_flag = test_df_total['y']
pred_flag = clf3.predict(test_df_total[train_df_total.columns[:-9]])

print("AUC score: {:<8.3f}".format(roc_auc_score(true_flag, pred_flag)))
AUC score: 0.804   

Check confusion matrix, AUC plot and other metrics

In [104]:
metrics = confusion_mat_plot(true_flag, pred_flag)
accuracy: 90.34 %
precision: 54.92 %
Recall: 67.77 %
specificity: 93.13 %
F-score: 60.67 %
In [105]:
gini = auc_plot(test_df_total[train_df_total.columns[:-9]], true_flag, clf3)

# record new metrics
model_comparison['Without Eng Feat'] = [gini, metrics[0], metrics[1], metrics[2], metrics[3], metrics[4]]
gini 87.409 %

Let's check the new feature importance (without engineered features).

In [106]:
feature_importance_df3 = pd.DataFrame()
feature_importance_df3["Feature"] = train_df_total.columns.to_list()[:-9]
feature_importance_df3["importance"] = clf3.feature_importances_

# plot results
plt.figure(figsize=(12,12))
sns.barplot(x="importance", y="Feature", data=feature_importance_df3.sort_values(by="importance",ascending=False))
plt.title('Features importance')
plt.tight_layout()
plt.show()

Model 4 - Top 5 features, without the engineered features

In [107]:
%%time
clf4 = xgb.XGBClassifier(**best_parameters)
top_5_feat_2 = feature_importance_df3.sort_values(by="importance",ascending=False)['Feature'].values[:5]
tr_val = (train_df_total[top_5_feat_2], target)
ts_val = (test_df_total[top_5_feat_2], test_df_total['y'])

clf4.fit(train_df_total[top_5_feat_2], target, eval_metric='auc', eval_set=[tr_val, ts_val])
[0]	validation_0-auc:0.791971	validation_1-auc:0.774297
[1]	validation_0-auc:0.7925	validation_1-auc:0.775093
[2]	validation_0-auc:0.792541	validation_1-auc:0.774931
[3]	validation_0-auc:0.792598	validation_1-auc:0.774884
[4]	validation_0-auc:0.792634	validation_1-auc:0.774935
[5]	validation_0-auc:0.792657	validation_1-auc:0.774895
[6]	validation_0-auc:0.792714	validation_1-auc:0.774961
[7]	validation_0-auc:0.792671	validation_1-auc:0.775159
[8]	validation_0-auc:0.792718	validation_1-auc:0.775097
[9]	validation_0-auc:0.79272	validation_1-auc:0.775086
[10]	validation_0-auc:0.792701	validation_1-auc:0.775091
[11]	validation_0-auc:0.792679	validation_1-auc:0.775218
[12]	validation_0-auc:0.792675	validation_1-auc:0.775225
[13]	validation_0-auc:0.792674	validation_1-auc:0.775225
[14]	validation_0-auc:0.792681	validation_1-auc:0.775282
[15]	validation_0-auc:0.792681	validation_1-auc:0.775297
[16]	validation_0-auc:0.792719	validation_1-auc:0.774903
[17]	validation_0-auc:0.792757	validation_1-auc:0.774989
[18]	validation_0-auc:0.792754	validation_1-auc:0.775053
[19]	validation_0-auc:0.792755	validation_1-auc:0.77491
[20]	validation_0-auc:0.792751	validation_1-auc:0.774922
[21]	validation_0-auc:0.792744	validation_1-auc:0.774896
[22]	validation_0-auc:0.792724	validation_1-auc:0.774826
[23]	validation_0-auc:0.792699	validation_1-auc:0.775011
[24]	validation_0-auc:0.792751	validation_1-auc:0.774869
[25]	validation_0-auc:0.792709	validation_1-auc:0.775048
[26]	validation_0-auc:0.792758	validation_1-auc:0.774925
[27]	validation_0-auc:0.792737	validation_1-auc:0.7746
[28]	validation_0-auc:0.792719	validation_1-auc:0.774961
[29]	validation_0-auc:0.792723	validation_1-auc:0.774983
[30]	validation_0-auc:0.792774	validation_1-auc:0.774836
[31]	validation_0-auc:0.792772	validation_1-auc:0.774892
[32]	validation_0-auc:0.792767	validation_1-auc:0.774938
[33]	validation_0-auc:0.792724	validation_1-auc:0.77507
[34]	validation_0-auc:0.792711	validation_1-auc:0.775063
[35]	validation_0-auc:0.79271	validation_1-auc:0.775056
[36]	validation_0-auc:0.792745	validation_1-auc:0.774972
[37]	validation_0-auc:0.792738	validation_1-auc:0.774964
[38]	validation_0-auc:0.792763	validation_1-auc:0.774927
[39]	validation_0-auc:0.792771	validation_1-auc:0.775087
[40]	validation_0-auc:0.792761	validation_1-auc:0.775143
[41]	validation_0-auc:0.792772	validation_1-auc:0.775106
[42]	validation_0-auc:0.79277	validation_1-auc:0.774939
[43]	validation_0-auc:0.792699	validation_1-auc:0.774859
[44]	validation_0-auc:0.792726	validation_1-auc:0.775021
[45]	validation_0-auc:0.792726	validation_1-auc:0.775022
[46]	validation_0-auc:0.792723	validation_1-auc:0.775042
[47]	validation_0-auc:0.792707	validation_1-auc:0.775049
[48]	validation_0-auc:0.792754	validation_1-auc:0.77494
[49]	validation_0-auc:0.792667	validation_1-auc:0.774881
[50]	validation_0-auc:0.792695	validation_1-auc:0.774973
[51]	validation_0-auc:0.792709	validation_1-auc:0.775055
[52]	validation_0-auc:0.792714	validation_1-auc:0.775041
[53]	validation_0-auc:0.792737	validation_1-auc:0.775014
[54]	validation_0-auc:0.792727	validation_1-auc:0.775017
[55]	validation_0-auc:0.792717	validation_1-auc:0.774943
[56]	validation_0-auc:0.792729	validation_1-auc:0.774993
[57]	validation_0-auc:0.792697	validation_1-auc:0.774716
[58]	validation_0-auc:0.79273	validation_1-auc:0.774991
[59]	validation_0-auc:0.792728	validation_1-auc:0.774998
[60]	validation_0-auc:0.792701	validation_1-auc:0.774841
[61]	validation_0-auc:0.792721	validation_1-auc:0.7748
[62]	validation_0-auc:0.792732	validation_1-auc:0.774894
[63]	validation_0-auc:0.792736	validation_1-auc:0.774891
[64]	validation_0-auc:0.792734	validation_1-auc:0.774852
[65]	validation_0-auc:0.792737	validation_1-auc:0.775008
[66]	validation_0-auc:0.792746	validation_1-auc:0.77501
[67]	validation_0-auc:0.792745	validation_1-auc:0.775017
[68]	validation_0-auc:0.792746	validation_1-auc:0.775024
[69]	validation_0-auc:0.792748	validation_1-auc:0.775042
[70]	validation_0-auc:0.792785	validation_1-auc:0.77491
[71]	validation_0-auc:0.792785	validation_1-auc:0.774921
[72]	validation_0-auc:0.792775	validation_1-auc:0.775004
[73]	validation_0-auc:0.792774	validation_1-auc:0.774991
[74]	validation_0-auc:0.792774	validation_1-auc:0.775008
[75]	validation_0-auc:0.79279	validation_1-auc:0.774994
[76]	validation_0-auc:0.792774	validation_1-auc:0.775025
[77]	validation_0-auc:0.79277	validation_1-auc:0.775107
[78]	validation_0-auc:0.792774	validation_1-auc:0.775109
[79]	validation_0-auc:0.792767	validation_1-auc:0.775093
[80]	validation_0-auc:0.792764	validation_1-auc:0.77514
[81]	validation_0-auc:0.792767	validation_1-auc:0.775114
[82]	validation_0-auc:0.792778	validation_1-auc:0.775094
[83]	validation_0-auc:0.792776	validation_1-auc:0.775126
[84]	validation_0-auc:0.792754	validation_1-auc:0.775142
[85]	validation_0-auc:0.792755	validation_1-auc:0.775122
[86]	validation_0-auc:0.792755	validation_1-auc:0.775116
[87]	validation_0-auc:0.792784	validation_1-auc:0.77511
[88]	validation_0-auc:0.792747	validation_1-auc:0.775218
[89]	validation_0-auc:0.792788	validation_1-auc:0.774882
[90]	validation_0-auc:0.792779	validation_1-auc:0.775082
[91]	validation_0-auc:0.792734	validation_1-auc:0.77526
[92]	validation_0-auc:0.792768	validation_1-auc:0.775106
[93]	validation_0-auc:0.792724	validation_1-auc:0.775248
[94]	validation_0-auc:0.792717	validation_1-auc:0.775086
[95]	validation_0-auc:0.792722	validation_1-auc:0.775098
[96]	validation_0-auc:0.792726	validation_1-auc:0.775121
[97]	validation_0-auc:0.792788	validation_1-auc:0.774954
[98]	validation_0-auc:0.792744	validation_1-auc:0.775068
[99]	validation_0-auc:0.792742	validation_1-auc:0.775067
[100]	validation_0-auc:0.792731	validation_1-auc:0.775001
[101]	validation_0-auc:0.792721	validation_1-auc:0.774854
[102]	validation_0-auc:0.792762	validation_1-auc:0.774734
[103]	validation_0-auc:0.792744	validation_1-auc:0.774628
[104]	validation_0-auc:0.792777	validation_1-auc:0.774884
[105]	validation_0-auc:0.792733	validation_1-auc:0.774997
[106]	validation_0-auc:0.792738	validation_1-auc:0.774667
[107]	validation_0-auc:0.792694	validation_1-auc:0.774858
[108]	validation_0-auc:0.792763	validation_1-auc:0.774684
[109]	validation_0-auc:0.792752	validation_1-auc:0.774654
[110]	validation_0-auc:0.792767	validation_1-auc:0.774658
[111]	validation_0-auc:0.792711	validation_1-auc:0.774832
[112]	validation_0-auc:0.792717	validation_1-auc:0.774872
[113]	validation_0-auc:0.792735	validation_1-auc:0.775064
[114]	validation_0-auc:0.792731	validation_1-auc:0.775038
[115]	validation_0-auc:0.792744	validation_1-auc:0.775086
[116]	validation_0-auc:0.79272	validation_1-auc:0.774863
[117]	validation_0-auc:0.792744	validation_1-auc:0.775051
[118]	validation_0-auc:0.792766	validation_1-auc:0.774733
[119]	validation_0-auc:0.79279	validation_1-auc:0.774954
[120]	validation_0-auc:0.792788	validation_1-auc:0.774916
[121]	validation_0-auc:0.792787	validation_1-auc:0.774855
[122]	validation_0-auc:0.792778	validation_1-auc:0.775014
[123]	validation_0-auc:0.792781	validation_1-auc:0.774998
[124]	validation_0-auc:0.792788	validation_1-auc:0.774924
[125]	validation_0-auc:0.79278	validation_1-auc:0.774934
[126]	validation_0-auc:0.792785	validation_1-auc:0.77491
[127]	validation_0-auc:0.792772	validation_1-auc:0.774907
[128]	validation_0-auc:0.792792	validation_1-auc:0.774926
[129]	validation_0-auc:0.792783	validation_1-auc:0.774937
[130]	validation_0-auc:0.792787	validation_1-auc:0.774866
[131]	validation_0-auc:0.792784	validation_1-auc:0.774754
[132]	validation_0-auc:0.792787	validation_1-auc:0.774753
[133]	validation_0-auc:0.792769	validation_1-auc:0.774851
[134]	validation_0-auc:0.792766	validation_1-auc:0.774814
[135]	validation_0-auc:0.792766	validation_1-auc:0.774792
[136]	validation_0-auc:0.792776	validation_1-auc:0.774748
[137]	validation_0-auc:0.792781	validation_1-auc:0.774907
[138]	validation_0-auc:0.792781	validation_1-auc:0.774892
[139]	validation_0-auc:0.792781	validation_1-auc:0.774897
[140]	validation_0-auc:0.792782	validation_1-auc:0.774938
[141]	validation_0-auc:0.79278	validation_1-auc:0.774934
[142]	validation_0-auc:0.792786	validation_1-auc:0.77491
[143]	validation_0-auc:0.792789	validation_1-auc:0.774933
[144]	validation_0-auc:0.79279	validation_1-auc:0.774933
[145]	validation_0-auc:0.792794	validation_1-auc:0.775008
[146]	validation_0-auc:0.792797	validation_1-auc:0.774961
[147]	validation_0-auc:0.792797	validation_1-auc:0.77496
[148]	validation_0-auc:0.792798	validation_1-auc:0.775064
[149]	validation_0-auc:0.792794	validation_1-auc:0.775071
[150]	validation_0-auc:0.792786	validation_1-auc:0.775055
[151]	validation_0-auc:0.792786	validation_1-auc:0.775045
[152]	validation_0-auc:0.792779	validation_1-auc:0.775047
[153]	validation_0-auc:0.792785	validation_1-auc:0.774967
[154]	validation_0-auc:0.792789	validation_1-auc:0.774923
[155]	validation_0-auc:0.792782	validation_1-auc:0.774819
[156]	validation_0-auc:0.792787	validation_1-auc:0.774846
[157]	validation_0-auc:0.792782	validation_1-auc:0.774846
[158]	validation_0-auc:0.792782	validation_1-auc:0.774839
[159]	validation_0-auc:0.792781	validation_1-auc:0.774864
[160]	validation_0-auc:0.792783	validation_1-auc:0.774843
[161]	validation_0-auc:0.792788	validation_1-auc:0.774975
[162]	validation_0-auc:0.792785	validation_1-auc:0.774792
[163]	validation_0-auc:0.792782	validation_1-auc:0.774875
[164]	validation_0-auc:0.792782	validation_1-auc:0.774865
[165]	validation_0-auc:0.792785	validation_1-auc:0.774948
[166]	validation_0-auc:0.792787	validation_1-auc:0.774881
[167]	validation_0-auc:0.792788	validation_1-auc:0.774855
[168]	validation_0-auc:0.792787	validation_1-auc:0.774862
[169]	validation_0-auc:0.792793	validation_1-auc:0.775002
[170]	validation_0-auc:0.792791	validation_1-auc:0.774895
[171]	validation_0-auc:0.792791	validation_1-auc:0.774857
[172]	validation_0-auc:0.792794	validation_1-auc:0.775047
[173]	validation_0-auc:0.792798	validation_1-auc:0.775006
[174]	validation_0-auc:0.79279	validation_1-auc:0.774882
[175]	validation_0-auc:0.792793	validation_1-auc:0.774883
[176]	validation_0-auc:0.792793	validation_1-auc:0.77487
[177]	validation_0-auc:0.792786	validation_1-auc:0.774894
[178]	validation_0-auc:0.79278	validation_1-auc:0.774957
[179]	validation_0-auc:0.792786	validation_1-auc:0.774928
[180]	validation_0-auc:0.792782	validation_1-auc:0.775046
[181]	validation_0-auc:0.792781	validation_1-auc:0.775057
[182]	validation_0-auc:0.792774	validation_1-auc:0.775083
[183]	validation_0-auc:0.79276	validation_1-auc:0.775061
[184]	validation_0-auc:0.79276	validation_1-auc:0.775062
[185]	validation_0-auc:0.792773	validation_1-auc:0.775078
[186]	validation_0-auc:0.792775	validation_1-auc:0.775061
[187]	validation_0-auc:0.792775	validation_1-auc:0.775065
[188]	validation_0-auc:0.79277	validation_1-auc:0.775116
[189]	validation_0-auc:0.792726	validation_1-auc:0.775101
[190]	validation_0-auc:0.792762	validation_1-auc:0.774946
[191]	validation_0-auc:0.792764	validation_1-auc:0.774948
[192]	validation_0-auc:0.792765	validation_1-auc:0.774919
[193]	validation_0-auc:0.79277	validation_1-auc:0.774966
[194]	validation_0-auc:0.792727	validation_1-auc:0.775077
[195]	validation_0-auc:0.792735	validation_1-auc:0.775202
[196]	validation_0-auc:0.792783	validation_1-auc:0.775134
[197]	validation_0-auc:0.792785	validation_1-auc:0.775087
[198]	validation_0-auc:0.792793	validation_1-auc:0.774924
[199]	validation_0-auc:0.792773	validation_1-auc:0.774887
[200]	validation_0-auc:0.792704	validation_1-auc:0.774843
[201]	validation_0-auc:0.79271	validation_1-auc:0.774842
[202]	validation_0-auc:0.792745	validation_1-auc:0.775029
[203]	validation_0-auc:0.792704	validation_1-auc:0.774812
[204]	validation_0-auc:0.792712	validation_1-auc:0.774823
[205]	validation_0-auc:0.79271	validation_1-auc:0.774817
[206]	validation_0-auc:0.792719	validation_1-auc:0.774828
[207]	validation_0-auc:0.792716	validation_1-auc:0.774839
[208]	validation_0-auc:0.792725	validation_1-auc:0.775047
[209]	validation_0-auc:0.792739	validation_1-auc:0.775059
[210]	validation_0-auc:0.792743	validation_1-auc:0.775069
[211]	validation_0-auc:0.792738	validation_1-auc:0.775069
[212]	validation_0-auc:0.792737	validation_1-auc:0.775036
[213]	validation_0-auc:0.792746	validation_1-auc:0.775045
[214]	validation_0-auc:0.792739	validation_1-auc:0.775108
[215]	validation_0-auc:0.792744	validation_1-auc:0.77505
[216]	validation_0-auc:0.792744	validation_1-auc:0.77507
[217]	validation_0-auc:0.792741	validation_1-auc:0.775085
[218]	validation_0-auc:0.792722	validation_1-auc:0.7748
[219]	validation_0-auc:0.792759	validation_1-auc:0.774689
[220]	validation_0-auc:0.79276	validation_1-auc:0.774717
[221]	validation_0-auc:0.792766	validation_1-auc:0.774633
[222]	validation_0-auc:0.792759	validation_1-auc:0.77453
[223]	validation_0-auc:0.792738	validation_1-auc:0.774448
[224]	validation_0-auc:0.792708	validation_1-auc:0.774808
[225]	validation_0-auc:0.792702	validation_1-auc:0.774799
[226]	validation_0-auc:0.792719	validation_1-auc:0.774672
[227]	validation_0-auc:0.79274	validation_1-auc:0.774865
[228]	validation_0-auc:0.792773	validation_1-auc:0.774609
[229]	validation_0-auc:0.792735	validation_1-auc:0.774869
[230]	validation_0-auc:0.792781	validation_1-auc:0.774728
[231]	validation_0-auc:0.792758	validation_1-auc:0.774518
[232]	validation_0-auc:0.792763	validation_1-auc:0.774511
[233]	validation_0-auc:0.792786	validation_1-auc:0.774742
[234]	validation_0-auc:0.792784	validation_1-auc:0.774724
[235]	validation_0-auc:0.79276	validation_1-auc:0.774545
[236]	validation_0-auc:0.792785	validation_1-auc:0.774753
[237]	validation_0-auc:0.792785	validation_1-auc:0.774724
[238]	validation_0-auc:0.792773	validation_1-auc:0.77474
[239]	validation_0-auc:0.792772	validation_1-auc:0.774724
[240]	validation_0-auc:0.792723	validation_1-auc:0.774853
[241]	validation_0-auc:0.792743	validation_1-auc:0.774908
[242]	validation_0-auc:0.792741	validation_1-auc:0.774901
[243]	validation_0-auc:0.79273	validation_1-auc:0.774896
[244]	validation_0-auc:0.792716	validation_1-auc:0.774881
[245]	validation_0-auc:0.792764	validation_1-auc:0.774738
[246]	validation_0-auc:0.792767	validation_1-auc:0.774772
[247]	validation_0-auc:0.792778	validation_1-auc:0.774748
[248]	validation_0-auc:0.792722	validation_1-auc:0.774895
[249]	validation_0-auc:0.79272	validation_1-auc:0.77489
[250]	validation_0-auc:0.792732	validation_1-auc:0.774889
[251]	validation_0-auc:0.792737	validation_1-auc:0.774903
[252]	validation_0-auc:0.792733	validation_1-auc:0.77489
[253]	validation_0-auc:0.792714	validation_1-auc:0.774751
[254]	validation_0-auc:0.792725	validation_1-auc:0.774846
[255]	validation_0-auc:0.792734	validation_1-auc:0.774952
[256]	validation_0-auc:0.79274	validation_1-auc:0.774978
[257]	validation_0-auc:0.792746	validation_1-auc:0.775205
[258]	validation_0-auc:0.792734	validation_1-auc:0.775155
[259]	validation_0-auc:0.792728	validation_1-auc:0.775257
[260]	validation_0-auc:0.792704	validation_1-auc:0.775199
[261]	validation_0-auc:0.792715	validation_1-auc:0.775202
[262]	validation_0-auc:0.792702	validation_1-auc:0.775168
[263]	validation_0-auc:0.792712	validation_1-auc:0.775225
[264]	validation_0-auc:0.79272	validation_1-auc:0.775005
[265]	validation_0-auc:0.792757	validation_1-auc:0.774804
[266]	validation_0-auc:0.792726	validation_1-auc:0.774984
[267]	validation_0-auc:0.792723	validation_1-auc:0.774981
[268]	validation_0-auc:0.792724	validation_1-auc:0.774961
[269]	validation_0-auc:0.792727	validation_1-auc:0.774977
[270]	validation_0-auc:0.792717	validation_1-auc:0.774898
[271]	validation_0-auc:0.792739	validation_1-auc:0.775009
[272]	validation_0-auc:0.792784	validation_1-auc:0.774862
[273]	validation_0-auc:0.792787	validation_1-auc:0.774858
[274]	validation_0-auc:0.79279	validation_1-auc:0.775042
[275]	validation_0-auc:0.792786	validation_1-auc:0.775043
[276]	validation_0-auc:0.792785	validation_1-auc:0.775001
[277]	validation_0-auc:0.792778	validation_1-auc:0.774991
[278]	validation_0-auc:0.792772	validation_1-auc:0.774982
[279]	validation_0-auc:0.792766	validation_1-auc:0.775072
[280]	validation_0-auc:0.792769	validation_1-auc:0.775095
[281]	validation_0-auc:0.792777	validation_1-auc:0.774993
[282]	validation_0-auc:0.792767	validation_1-auc:0.775015
[283]	validation_0-auc:0.792771	validation_1-auc:0.774933
[284]	validation_0-auc:0.792769	validation_1-auc:0.774796
[285]	validation_0-auc:0.792759	validation_1-auc:0.774817
[286]	validation_0-auc:0.792759	validation_1-auc:0.774816
[287]	validation_0-auc:0.792764	validation_1-auc:0.774849
[288]	validation_0-auc:0.792764	validation_1-auc:0.774849
[289]	validation_0-auc:0.792766	validation_1-auc:0.774987
[290]	validation_0-auc:0.792769	validation_1-auc:0.774963
[291]	validation_0-auc:0.792764	validation_1-auc:0.774842
[292]	validation_0-auc:0.792766	validation_1-auc:0.774841
[293]	validation_0-auc:0.792769	validation_1-auc:0.774846
[294]	validation_0-auc:0.792768	validation_1-auc:0.774837
[295]	validation_0-auc:0.792756	validation_1-auc:0.774823
[296]	validation_0-auc:0.792766	validation_1-auc:0.774873
[297]	validation_0-auc:0.792763	validation_1-auc:0.774862
[298]	validation_0-auc:0.792777	validation_1-auc:0.774887
[299]	validation_0-auc:0.792778	validation_1-auc:0.774904
[300]	validation_0-auc:0.792781	validation_1-auc:0.7749
[301]	validation_0-auc:0.792775	validation_1-auc:0.774878
[302]	validation_0-auc:0.79273	validation_1-auc:0.775013
[303]	validation_0-auc:0.792773	validation_1-auc:0.774861
[304]	validation_0-auc:0.792776	validation_1-auc:0.774867
[305]	validation_0-auc:0.792731	validation_1-auc:0.775055
[306]	validation_0-auc:0.79273	validation_1-auc:0.774977
[307]	validation_0-auc:0.792726	validation_1-auc:0.775012
[308]	validation_0-auc:0.792734	validation_1-auc:0.775008
[309]	validation_0-auc:0.792721	validation_1-auc:0.774958
[310]	validation_0-auc:0.792734	validation_1-auc:0.775007
[311]	validation_0-auc:0.792731	validation_1-auc:0.774857
[312]	validation_0-auc:0.79272	validation_1-auc:0.774924
[313]	validation_0-auc:0.792772	validation_1-auc:0.774968
[314]	validation_0-auc:0.792763	validation_1-auc:0.774782
[315]	validation_0-auc:0.792776	validation_1-auc:0.774909
[316]	validation_0-auc:0.792774	validation_1-auc:0.774773
[317]	validation_0-auc:0.792774	validation_1-auc:0.774988
[318]	validation_0-auc:0.792769	validation_1-auc:0.774845
[319]	validation_0-auc:0.792765	validation_1-auc:0.774877
[320]	validation_0-auc:0.792765	validation_1-auc:0.774868
[321]	validation_0-auc:0.79277	validation_1-auc:0.775066
[322]	validation_0-auc:0.792774	validation_1-auc:0.775021
[323]	validation_0-auc:0.792786	validation_1-auc:0.774974
[324]	validation_0-auc:0.792781	validation_1-auc:0.774837
[325]	validation_0-auc:0.792782	validation_1-auc:0.77482
[326]	validation_0-auc:0.792787	validation_1-auc:0.774949
[327]	validation_0-auc:0.792789	validation_1-auc:0.774972
[328]	validation_0-auc:0.792788	validation_1-auc:0.775
[329]	validation_0-auc:0.792791	validation_1-auc:0.774943
[330]	validation_0-auc:0.792793	validation_1-auc:0.77494
[331]	validation_0-auc:0.79279	validation_1-auc:0.774907
[332]	validation_0-auc:0.792786	validation_1-auc:0.774759
[333]	validation_0-auc:0.792791	validation_1-auc:0.774926
[334]	validation_0-auc:0.792786	validation_1-auc:0.774771
[335]	validation_0-auc:0.79275	validation_1-auc:0.77457
[336]	validation_0-auc:0.792739	validation_1-auc:0.774625
[337]	validation_0-auc:0.792749	validation_1-auc:0.774541
[338]	validation_0-auc:0.792781	validation_1-auc:0.774743
[339]	validation_0-auc:0.792758	validation_1-auc:0.774526
[340]	validation_0-auc:0.792778	validation_1-auc:0.774847
[341]	validation_0-auc:0.792757	validation_1-auc:0.774612
[342]	validation_0-auc:0.792759	validation_1-auc:0.774684
[343]	validation_0-auc:0.792768	validation_1-auc:0.774669
[344]	validation_0-auc:0.792782	validation_1-auc:0.774738
[345]	validation_0-auc:0.792774	validation_1-auc:0.774738
[346]	validation_0-auc:0.792782	validation_1-auc:0.774803
[347]	validation_0-auc:0.792781	validation_1-auc:0.774806
[348]	validation_0-auc:0.792747	validation_1-auc:0.77459
[349]	validation_0-auc:0.792752	validation_1-auc:0.774549
[350]	validation_0-auc:0.792759	validation_1-auc:0.774559
[351]	validation_0-auc:0.792773	validation_1-auc:0.774641
[352]	validation_0-auc:0.79279	validation_1-auc:0.774824
[353]	validation_0-auc:0.792795	validation_1-auc:0.77496
[354]	validation_0-auc:0.792782	validation_1-auc:0.774724
[355]	validation_0-auc:0.792786	validation_1-auc:0.774739
[356]	validation_0-auc:0.792782	validation_1-auc:0.77476
[357]	validation_0-auc:0.792785	validation_1-auc:0.774757
[358]	validation_0-auc:0.79278	validation_1-auc:0.77479
[359]	validation_0-auc:0.792774	validation_1-auc:0.774854
[360]	validation_0-auc:0.792745	validation_1-auc:0.774534
[361]	validation_0-auc:0.792758	validation_1-auc:0.774549
[362]	validation_0-auc:0.792751	validation_1-auc:0.774571
[363]	validation_0-auc:0.792753	validation_1-auc:0.774558
[364]	validation_0-auc:0.792782	validation_1-auc:0.774799
[365]	validation_0-auc:0.79278	validation_1-auc:0.77473
[366]	validation_0-auc:0.792778	validation_1-auc:0.774818
[367]	validation_0-auc:0.792772	validation_1-auc:0.77479
[368]	validation_0-auc:0.792747	validation_1-auc:0.774582
[369]	validation_0-auc:0.792771	validation_1-auc:0.774776
[370]	validation_0-auc:0.79278	validation_1-auc:0.774713
[371]	validation_0-auc:0.792737	validation_1-auc:0.77494
[372]	validation_0-auc:0.792722	validation_1-auc:0.774854
[373]	validation_0-auc:0.792714	validation_1-auc:0.774953
[374]	validation_0-auc:0.792717	validation_1-auc:0.774955
[375]	validation_0-auc:0.792764	validation_1-auc:0.774947
[376]	validation_0-auc:0.792769	validation_1-auc:0.774917
[377]	validation_0-auc:0.792767	validation_1-auc:0.774934
[378]	validation_0-auc:0.792773	validation_1-auc:0.774891
[379]	validation_0-auc:0.792784	validation_1-auc:0.774856
[380]	validation_0-auc:0.792789	validation_1-auc:0.775009
[381]	validation_0-auc:0.792791	validation_1-auc:0.774998
[382]	validation_0-auc:0.792793	validation_1-auc:0.775013
[383]	validation_0-auc:0.792792	validation_1-auc:0.774998
[384]	validation_0-auc:0.79279	validation_1-auc:0.774921
[385]	validation_0-auc:0.792786	validation_1-auc:0.774955
[386]	validation_0-auc:0.792786	validation_1-auc:0.774952
[387]	validation_0-auc:0.792791	validation_1-auc:0.774919
[388]	validation_0-auc:0.792791	validation_1-auc:0.774916
[389]	validation_0-auc:0.792794	validation_1-auc:0.774932
[390]	validation_0-auc:0.792795	validation_1-auc:0.77494
[391]	validation_0-auc:0.792792	validation_1-auc:0.774929
[392]	validation_0-auc:0.792784	validation_1-auc:0.774876
[393]	validation_0-auc:0.79279	validation_1-auc:0.77492
[394]	validation_0-auc:0.792795	validation_1-auc:0.774932
[395]	validation_0-auc:0.792752	validation_1-auc:0.775083
[396]	validation_0-auc:0.792752	validation_1-auc:0.775069
[397]	validation_0-auc:0.792751	validation_1-auc:0.775093
[398]	validation_0-auc:0.792744	validation_1-auc:0.775157
[399]	validation_0-auc:0.792789	validation_1-auc:0.775008
[400]	validation_0-auc:0.792789	validation_1-auc:0.775012
[401]	validation_0-auc:0.792795	validation_1-auc:0.774996
[402]	validation_0-auc:0.792792	validation_1-auc:0.774924
[403]	validation_0-auc:0.792786	validation_1-auc:0.774928
[404]	validation_0-auc:0.79279	validation_1-auc:0.774965
[405]	validation_0-auc:0.792787	validation_1-auc:0.774968
[406]	validation_0-auc:0.792785	validation_1-auc:0.775083
[407]	validation_0-auc:0.792778	validation_1-auc:0.774979
[408]	validation_0-auc:0.792778	validation_1-auc:0.774979
[409]	validation_0-auc:0.792777	validation_1-auc:0.774975
[410]	validation_0-auc:0.792731	validation_1-auc:0.775092
[411]	validation_0-auc:0.792733	validation_1-auc:0.775098
[412]	validation_0-auc:0.792733	validation_1-auc:0.775071
[413]	validation_0-auc:0.792727	validation_1-auc:0.77505
[414]	validation_0-auc:0.792732	validation_1-auc:0.775089
[415]	validation_0-auc:0.792734	validation_1-auc:0.77505
[416]	validation_0-auc:0.792736	validation_1-auc:0.774943
[417]	validation_0-auc:0.792739	validation_1-auc:0.774934
[418]	validation_0-auc:0.79274	validation_1-auc:0.774985
[419]	validation_0-auc:0.792786	validation_1-auc:0.774775
[420]	validation_0-auc:0.792781	validation_1-auc:0.774944
[421]	validation_0-auc:0.792783	validation_1-auc:0.77491
[422]	validation_0-auc:0.792774	validation_1-auc:0.774751
[423]	validation_0-auc:0.792774	validation_1-auc:0.774784
[424]	validation_0-auc:0.792747	validation_1-auc:0.774573
[425]	validation_0-auc:0.792771	validation_1-auc:0.774793
[426]	validation_0-auc:0.792777	validation_1-auc:0.774988
[427]	validation_0-auc:0.792779	validation_1-auc:0.77499
[428]	validation_0-auc:0.792741	validation_1-auc:0.775112
[429]	validation_0-auc:0.792776	validation_1-auc:0.774978
[430]	validation_0-auc:0.792778	validation_1-auc:0.774891
[431]	validation_0-auc:0.792773	validation_1-auc:0.774906
[432]	validation_0-auc:0.792785	validation_1-auc:0.774884
[433]	validation_0-auc:0.792778	validation_1-auc:0.775028
[434]	validation_0-auc:0.792789	validation_1-auc:0.774898
[435]	validation_0-auc:0.792787	validation_1-auc:0.774973
[436]	validation_0-auc:0.792792	validation_1-auc:0.775001
[437]	validation_0-auc:0.792768	validation_1-auc:0.775085
[438]	validation_0-auc:0.792777	validation_1-auc:0.774977
[439]	validation_0-auc:0.792761	validation_1-auc:0.774933
[440]	validation_0-auc:0.792778	validation_1-auc:0.77498
[441]	validation_0-auc:0.792781	validation_1-auc:0.775022
[442]	validation_0-auc:0.79276	validation_1-auc:0.774851
[443]	validation_0-auc:0.792775	validation_1-auc:0.775058
[444]	validation_0-auc:0.792773	validation_1-auc:0.774952
[445]	validation_0-auc:0.792769	validation_1-auc:0.774921
[446]	validation_0-auc:0.79278	validation_1-auc:0.775091
[447]	validation_0-auc:0.792781	validation_1-auc:0.775068
[448]	validation_0-auc:0.792782	validation_1-auc:0.775068
[449]	validation_0-auc:0.792766	validation_1-auc:0.774864
[450]	validation_0-auc:0.79274	validation_1-auc:0.774738
[451]	validation_0-auc:0.792776	validation_1-auc:0.774936
[452]	validation_0-auc:0.792787	validation_1-auc:0.775078
[453]	validation_0-auc:0.792781	validation_1-auc:0.775053
[454]	validation_0-auc:0.792793	validation_1-auc:0.774985
[455]	validation_0-auc:0.79279	validation_1-auc:0.774992
[456]	validation_0-auc:0.792796	validation_1-auc:0.774995
[457]	validation_0-auc:0.792795	validation_1-auc:0.774996
[458]	validation_0-auc:0.792794	validation_1-auc:0.775007
[459]	validation_0-auc:0.792789	validation_1-auc:0.775033
[460]	validation_0-auc:0.792792	validation_1-auc:0.775038
[461]	validation_0-auc:0.792786	validation_1-auc:0.77507
[462]	validation_0-auc:0.792796	validation_1-auc:0.775081
[463]	validation_0-auc:0.792795	validation_1-auc:0.775071
[464]	validation_0-auc:0.792792	validation_1-auc:0.775089
[465]	validation_0-auc:0.792789	validation_1-auc:0.775073
[466]	validation_0-auc:0.792789	validation_1-auc:0.775065
[467]	validation_0-auc:0.792775	validation_1-auc:0.775065
[468]	validation_0-auc:0.792792	validation_1-auc:0.775084
[469]	validation_0-auc:0.79279	validation_1-auc:0.775027
[470]	validation_0-auc:0.79279	validation_1-auc:0.775074
[471]	validation_0-auc:0.792787	validation_1-auc:0.775043
[472]	validation_0-auc:0.792731	validation_1-auc:0.775116
[473]	validation_0-auc:0.792749	validation_1-auc:0.77506
[474]	validation_0-auc:0.792796	validation_1-auc:0.774934
[475]	validation_0-auc:0.792767	validation_1-auc:0.775056
[476]	validation_0-auc:0.792786	validation_1-auc:0.774911
[477]	validation_0-auc:0.792736	validation_1-auc:0.775069
[478]	validation_0-auc:0.792773	validation_1-auc:0.774889
[479]	validation_0-auc:0.792728	validation_1-auc:0.775029
[480]	validation_0-auc:0.792721	validation_1-auc:0.775078
[481]	validation_0-auc:0.792724	validation_1-auc:0.775109
[482]	validation_0-auc:0.792762	validation_1-auc:0.774802
[483]	validation_0-auc:0.792767	validation_1-auc:0.774781
[484]	validation_0-auc:0.792774	validation_1-auc:0.774878
[485]	validation_0-auc:0.792776	validation_1-auc:0.774829
[486]	validation_0-auc:0.792779	validation_1-auc:0.774808
[487]	validation_0-auc:0.792779	validation_1-auc:0.774913
[488]	validation_0-auc:0.792784	validation_1-auc:0.774965
[489]	validation_0-auc:0.792733	validation_1-auc:0.775104
[490]	validation_0-auc:0.792731	validation_1-auc:0.775081
[491]	validation_0-auc:0.792732	validation_1-auc:0.774965
[492]	validation_0-auc:0.79274	validation_1-auc:0.774912
[493]	validation_0-auc:0.792738	validation_1-auc:0.774905
[494]	validation_0-auc:0.79274	validation_1-auc:0.774905
[495]	validation_0-auc:0.792743	validation_1-auc:0.774913
[496]	validation_0-auc:0.792735	validation_1-auc:0.775113
[497]	validation_0-auc:0.792775	validation_1-auc:0.774762
[498]	validation_0-auc:0.792789	validation_1-auc:0.774929
[499]	validation_0-auc:0.792741	validation_1-auc:0.775102
[500]	validation_0-auc:0.792741	validation_1-auc:0.774939
[501]	validation_0-auc:0.79275	validation_1-auc:0.775075
[502]	validation_0-auc:0.792787	validation_1-auc:0.774792
[503]	validation_0-auc:0.79278	validation_1-auc:0.774814
[504]	validation_0-auc:0.792779	validation_1-auc:0.774793
[505]	validation_0-auc:0.792781	validation_1-auc:0.774798
[506]	validation_0-auc:0.792786	validation_1-auc:0.774806
[507]	validation_0-auc:0.792785	validation_1-auc:0.774808
[508]	validation_0-auc:0.792797	validation_1-auc:0.775031
[509]	validation_0-auc:0.792793	validation_1-auc:0.775
[510]	validation_0-auc:0.792792	validation_1-auc:0.774986
[511]	validation_0-auc:0.792794	validation_1-auc:0.774898
[512]	validation_0-auc:0.792796	validation_1-auc:0.774967
[513]	validation_0-auc:0.792738	validation_1-auc:0.775031
[514]	validation_0-auc:0.792744	validation_1-auc:0.774968
[515]	validation_0-auc:0.792721	validation_1-auc:0.774763
[516]	validation_0-auc:0.792766	validation_1-auc:0.774623
[517]	validation_0-auc:0.792763	validation_1-auc:0.774552
[518]	validation_0-auc:0.792715	validation_1-auc:0.774719
[519]	validation_0-auc:0.792705	validation_1-auc:0.774699
[520]	validation_0-auc:0.792717	validation_1-auc:0.774805
[521]	validation_0-auc:0.792718	validation_1-auc:0.774788
[522]	validation_0-auc:0.792762	validation_1-auc:0.774662
[523]	validation_0-auc:0.792768	validation_1-auc:0.774652
[524]	validation_0-auc:0.792767	validation_1-auc:0.774649
[525]	validation_0-auc:0.792776	validation_1-auc:0.774732
[526]	validation_0-auc:0.792766	validation_1-auc:0.774647
[527]	validation_0-auc:0.792768	validation_1-auc:0.774641
[528]	validation_0-auc:0.79277	validation_1-auc:0.77475
[529]	validation_0-auc:0.792785	validation_1-auc:0.774882
[530]	validation_0-auc:0.792768	validation_1-auc:0.774649
[531]	validation_0-auc:0.792768	validation_1-auc:0.774646
[532]	validation_0-auc:0.792761	validation_1-auc:0.77465
[533]	validation_0-auc:0.792759	validation_1-auc:0.774721
[534]	validation_0-auc:0.792762	validation_1-auc:0.774709
[535]	validation_0-auc:0.792758	validation_1-auc:0.774709
[536]	validation_0-auc:0.792761	validation_1-auc:0.774701
[537]	validation_0-auc:0.792757	validation_1-auc:0.77471
[538]	validation_0-auc:0.792767	validation_1-auc:0.774795
[539]	validation_0-auc:0.792761	validation_1-auc:0.774786
[540]	validation_0-auc:0.792787	validation_1-auc:0.774913
[541]	validation_0-auc:0.792784	validation_1-auc:0.774896
[542]	validation_0-auc:0.792783	validation_1-auc:0.774854
[543]	validation_0-auc:0.792786	validation_1-auc:0.774802
[544]	validation_0-auc:0.79279	validation_1-auc:0.774873
[545]	validation_0-auc:0.792787	validation_1-auc:0.774867
[546]	validation_0-auc:0.792767	validation_1-auc:0.774642
[547]	validation_0-auc:0.792767	validation_1-auc:0.774656
[548]	validation_0-auc:0.792762	validation_1-auc:0.774636
[549]	validation_0-auc:0.792755	validation_1-auc:0.774639
[550]	validation_0-auc:0.792757	validation_1-auc:0.774711
[551]	validation_0-auc:0.792759	validation_1-auc:0.77465
[552]	validation_0-auc:0.792768	validation_1-auc:0.774734
[553]	validation_0-auc:0.792775	validation_1-auc:0.774837
[554]	validation_0-auc:0.792778	validation_1-auc:0.774858
[555]	validation_0-auc:0.792736	validation_1-auc:0.774989
[556]	validation_0-auc:0.792733	validation_1-auc:0.774992
[557]	validation_0-auc:0.792735	validation_1-auc:0.774982
[558]	validation_0-auc:0.792726	validation_1-auc:0.774985
[559]	validation_0-auc:0.792718	validation_1-auc:0.774921
[560]	validation_0-auc:0.792721	validation_1-auc:0.77485
[561]	validation_0-auc:0.792711	validation_1-auc:0.774918
[562]	validation_0-auc:0.792715	validation_1-auc:0.77489
[563]	validation_0-auc:0.792697	validation_1-auc:0.774887
[564]	validation_0-auc:0.792749	validation_1-auc:0.774877
[565]	validation_0-auc:0.792684	validation_1-auc:0.775069
[566]	validation_0-auc:0.792688	validation_1-auc:0.77506
[567]	validation_0-auc:0.792723	validation_1-auc:0.774941
[568]	validation_0-auc:0.792754	validation_1-auc:0.774845
[569]	validation_0-auc:0.792767	validation_1-auc:0.774817
[570]	validation_0-auc:0.792772	validation_1-auc:0.774806
[571]	validation_0-auc:0.792729	validation_1-auc:0.774939
[572]	validation_0-auc:0.792728	validation_1-auc:0.774936
[573]	validation_0-auc:0.792766	validation_1-auc:0.77477
[574]	validation_0-auc:0.792762	validation_1-auc:0.77477
[575]	validation_0-auc:0.792761	validation_1-auc:0.774753
[576]	validation_0-auc:0.792769	validation_1-auc:0.774779
[577]	validation_0-auc:0.792773	validation_1-auc:0.774764
[578]	validation_0-auc:0.792766	validation_1-auc:0.774802
[579]	validation_0-auc:0.792735	validation_1-auc:0.775154
[580]	validation_0-auc:0.79278	validation_1-auc:0.775024
[581]	validation_0-auc:0.792777	validation_1-auc:0.775
[582]	validation_0-auc:0.792788	validation_1-auc:0.77499
[583]	validation_0-auc:0.792788	validation_1-auc:0.774997
[584]	validation_0-auc:0.792782	validation_1-auc:0.774905
[585]	validation_0-auc:0.792785	validation_1-auc:0.774918
[586]	validation_0-auc:0.792738	validation_1-auc:0.774955
[587]	validation_0-auc:0.792713	validation_1-auc:0.774751
[588]	validation_0-auc:0.79278	validation_1-auc:0.774773
[589]	validation_0-auc:0.792723	validation_1-auc:0.774888
[590]	validation_0-auc:0.792726	validation_1-auc:0.774917
[591]	validation_0-auc:0.79273	validation_1-auc:0.774866
[592]	validation_0-auc:0.792745	validation_1-auc:0.774933
[593]	validation_0-auc:0.792743	validation_1-auc:0.774991
[594]	validation_0-auc:0.792744	validation_1-auc:0.774942
[595]	validation_0-auc:0.792742	validation_1-auc:0.775023
[596]	validation_0-auc:0.792741	validation_1-auc:0.775042
[597]	validation_0-auc:0.792708	validation_1-auc:0.774719
[598]	validation_0-auc:0.792709	validation_1-auc:0.774699
[599]	validation_0-auc:0.792707	validation_1-auc:0.774732
[600]	validation_0-auc:0.792734	validation_1-auc:0.774923
[601]	validation_0-auc:0.792731	validation_1-auc:0.774942
[602]	validation_0-auc:0.792728	validation_1-auc:0.774958
[603]	validation_0-auc:0.792752	validation_1-auc:0.774583
[604]	validation_0-auc:0.792779	validation_1-auc:0.774802
[605]	validation_0-auc:0.792758	validation_1-auc:0.774705
[606]	validation_0-auc:0.792753	validation_1-auc:0.774736
[607]	validation_0-auc:0.792776	validation_1-auc:0.774757
[608]	validation_0-auc:0.79277	validation_1-auc:0.77483
[609]	validation_0-auc:0.792769	validation_1-auc:0.774776
[610]	validation_0-auc:0.792764	validation_1-auc:0.775075
[611]	validation_0-auc:0.792773	validation_1-auc:0.774841
[612]	validation_0-auc:0.792738	validation_1-auc:0.775121
[613]	validation_0-auc:0.792743	validation_1-auc:0.774974
[614]	validation_0-auc:0.792724	validation_1-auc:0.774898
[615]	validation_0-auc:0.792732	validation_1-auc:0.774913
[616]	validation_0-auc:0.792742	validation_1-auc:0.775084
[617]	validation_0-auc:0.792743	validation_1-auc:0.775154
[618]	validation_0-auc:0.792743	validation_1-auc:0.775154
[619]	validation_0-auc:0.792741	validation_1-auc:0.77509
[620]	validation_0-auc:0.792741	validation_1-auc:0.775161
[621]	validation_0-auc:0.792742	validation_1-auc:0.775153
[622]	validation_0-auc:0.792738	validation_1-auc:0.775156
[623]	validation_0-auc:0.792735	validation_1-auc:0.775001
[624]	validation_0-auc:0.792731	validation_1-auc:0.775095
[625]	validation_0-auc:0.79273	validation_1-auc:0.775016
[626]	validation_0-auc:0.792732	validation_1-auc:0.775024
[627]	validation_0-auc:0.79273	validation_1-auc:0.775015
[628]	validation_0-auc:0.792721	validation_1-auc:0.775008
[629]	validation_0-auc:0.792734	validation_1-auc:0.77499
[630]	validation_0-auc:0.792725	validation_1-auc:0.774912
[631]	validation_0-auc:0.792724	validation_1-auc:0.774915
[632]	validation_0-auc:0.792732	validation_1-auc:0.774922
[633]	validation_0-auc:0.792746	validation_1-auc:0.774965
[634]	validation_0-auc:0.792734	validation_1-auc:0.774908
[635]	validation_0-auc:0.792731	validation_1-auc:0.774902
[636]	validation_0-auc:0.792746	validation_1-auc:0.775098
[637]	validation_0-auc:0.792781	validation_1-auc:0.774743
[638]	validation_0-auc:0.792776	validation_1-auc:0.774823
[639]	validation_0-auc:0.792771	validation_1-auc:0.775008
[640]	validation_0-auc:0.792781	validation_1-auc:0.774967
[641]	validation_0-auc:0.792767	validation_1-auc:0.774916
[642]	validation_0-auc:0.792764	validation_1-auc:0.774998
[643]	validation_0-auc:0.792781	validation_1-auc:0.775012
[644]	validation_0-auc:0.792782	validation_1-auc:0.774983
[645]	validation_0-auc:0.792787	validation_1-auc:0.774871
[646]	validation_0-auc:0.79279	validation_1-auc:0.774845
[647]	validation_0-auc:0.792791	validation_1-auc:0.77488
[648]	validation_0-auc:0.792786	validation_1-auc:0.774876
[649]	validation_0-auc:0.79279	validation_1-auc:0.774877
[650]	validation_0-auc:0.792746	validation_1-auc:0.775105
[651]	validation_0-auc:0.792745	validation_1-auc:0.775093
[652]	validation_0-auc:0.792752	validation_1-auc:0.775121
[653]	validation_0-auc:0.792731	validation_1-auc:0.775049
[654]	validation_0-auc:0.792746	validation_1-auc:0.775023
[655]	validation_0-auc:0.792747	validation_1-auc:0.775113
[656]	validation_0-auc:0.792746	validation_1-auc:0.775104
[657]	validation_0-auc:0.792746	validation_1-auc:0.775106
[658]	validation_0-auc:0.792747	validation_1-auc:0.775111
[659]	validation_0-auc:0.792745	validation_1-auc:0.775118
[660]	validation_0-auc:0.792729	validation_1-auc:0.774902
[661]	validation_0-auc:0.79273	validation_1-auc:0.774915
[662]	validation_0-auc:0.792734	validation_1-auc:0.774951
[663]	validation_0-auc:0.792737	validation_1-auc:0.775044
[664]	validation_0-auc:0.792736	validation_1-auc:0.775056
[665]	validation_0-auc:0.79275	validation_1-auc:0.7751
[666]	validation_0-auc:0.79275	validation_1-auc:0.775095
[667]	validation_0-auc:0.792742	validation_1-auc:0.775132
[668]	validation_0-auc:0.792746	validation_1-auc:0.775102
[669]	validation_0-auc:0.792745	validation_1-auc:0.775138
[670]	validation_0-auc:0.792747	validation_1-auc:0.775044
[671]	validation_0-auc:0.792748	validation_1-auc:0.775052
[672]	validation_0-auc:0.792743	validation_1-auc:0.775046
[673]	validation_0-auc:0.792743	validation_1-auc:0.775092
[674]	validation_0-auc:0.792745	validation_1-auc:0.775072
[675]	validation_0-auc:0.792746	validation_1-auc:0.775199
[676]	validation_0-auc:0.792791	validation_1-auc:0.775065
[677]	validation_0-auc:0.792746	validation_1-auc:0.775199
[678]	validation_0-auc:0.792744	validation_1-auc:0.775257
[679]	validation_0-auc:0.792741	validation_1-auc:0.775231
[680]	validation_0-auc:0.792739	validation_1-auc:0.775228
[681]	validation_0-auc:0.792745	validation_1-auc:0.77527
[682]	validation_0-auc:0.792742	validation_1-auc:0.775213
[683]	validation_0-auc:0.792782	validation_1-auc:0.775115
[684]	validation_0-auc:0.792738	validation_1-auc:0.775145
[685]	validation_0-auc:0.79274	validation_1-auc:0.775209
[686]	validation_0-auc:0.792737	validation_1-auc:0.775126
[687]	validation_0-auc:0.792738	validation_1-auc:0.775105
[688]	validation_0-auc:0.792711	validation_1-auc:0.775053
[689]	validation_0-auc:0.792709	validation_1-auc:0.775054
[690]	validation_0-auc:0.792719	validation_1-auc:0.775097
[691]	validation_0-auc:0.792728	validation_1-auc:0.775108
[692]	validation_0-auc:0.792703	validation_1-auc:0.775229
[693]	validation_0-auc:0.792717	validation_1-auc:0.77528
[694]	validation_0-auc:0.792739	validation_1-auc:0.775161
[695]	validation_0-auc:0.792738	validation_1-auc:0.775122
[696]	validation_0-auc:0.792783	validation_1-auc:0.774987
[697]	validation_0-auc:0.792737	validation_1-auc:0.775017
[698]	validation_0-auc:0.792732	validation_1-auc:0.775027
[699]	validation_0-auc:0.792739	validation_1-auc:0.775112
[700]	validation_0-auc:0.792743	validation_1-auc:0.775126
[701]	validation_0-auc:0.79274	validation_1-auc:0.775168
[702]	validation_0-auc:0.792748	validation_1-auc:0.775144
[703]	validation_0-auc:0.792753	validation_1-auc:0.775156
[704]	validation_0-auc:0.792741	validation_1-auc:0.775166
[705]	validation_0-auc:0.792745	validation_1-auc:0.775193
[706]	validation_0-auc:0.792734	validation_1-auc:0.775178
[707]	validation_0-auc:0.792784	validation_1-auc:0.775013
[708]	validation_0-auc:0.792797	validation_1-auc:0.775004
[709]	validation_0-auc:0.792748	validation_1-auc:0.774992
[710]	validation_0-auc:0.792732	validation_1-auc:0.774901
[711]	validation_0-auc:0.792745	validation_1-auc:0.774926
[712]	validation_0-auc:0.792744	validation_1-auc:0.774891
[713]	validation_0-auc:0.792744	validation_1-auc:0.774918
[714]	validation_0-auc:0.792744	validation_1-auc:0.774966
[715]	validation_0-auc:0.792744	validation_1-auc:0.774966
[716]	validation_0-auc:0.792744	validation_1-auc:0.774888
[717]	validation_0-auc:0.792738	validation_1-auc:0.774955
[718]	validation_0-auc:0.792743	validation_1-auc:0.774977
[719]	validation_0-auc:0.792737	validation_1-auc:0.774988
[720]	validation_0-auc:0.792741	validation_1-auc:0.775188
[721]	validation_0-auc:0.792735	validation_1-auc:0.775215
[722]	validation_0-auc:0.792747	validation_1-auc:0.775128
[723]	validation_0-auc:0.792742	validation_1-auc:0.775207
[724]	validation_0-auc:0.792735	validation_1-auc:0.775153
[725]	validation_0-auc:0.792733	validation_1-auc:0.775089
[726]	validation_0-auc:0.792732	validation_1-auc:0.775122
[727]	validation_0-auc:0.792736	validation_1-auc:0.77513
[728]	validation_0-auc:0.792741	validation_1-auc:0.775111
[729]	validation_0-auc:0.792731	validation_1-auc:0.775143
[730]	validation_0-auc:0.792733	validation_1-auc:0.775138
[731]	validation_0-auc:0.792778	validation_1-auc:0.774922
[732]	validation_0-auc:0.792733	validation_1-auc:0.775215
[733]	validation_0-auc:0.792738	validation_1-auc:0.775211
[734]	validation_0-auc:0.792737	validation_1-auc:0.77505
[735]	validation_0-auc:0.79274	validation_1-auc:0.775053
[736]	validation_0-auc:0.792789	validation_1-auc:0.774943
[737]	validation_0-auc:0.792739	validation_1-auc:0.77508
[738]	validation_0-auc:0.792746	validation_1-auc:0.775048
[739]	validation_0-auc:0.79275	validation_1-auc:0.775181
[740]	validation_0-auc:0.792734	validation_1-auc:0.775037
[741]	validation_0-auc:0.792735	validation_1-auc:0.775073
[742]	validation_0-auc:0.792733	validation_1-auc:0.775128
[743]	validation_0-auc:0.79273	validation_1-auc:0.775105
[744]	validation_0-auc:0.792709	validation_1-auc:0.774929
[745]	validation_0-auc:0.792693	validation_1-auc:0.774901
[746]	validation_0-auc:0.792728	validation_1-auc:0.775113
[747]	validation_0-auc:0.792729	validation_1-auc:0.775125
[748]	validation_0-auc:0.792724	validation_1-auc:0.775315
[749]	validation_0-auc:0.792732	validation_1-auc:0.775286
[750]	validation_0-auc:0.79273	validation_1-auc:0.775287
[751]	validation_0-auc:0.792729	validation_1-auc:0.775264
[752]	validation_0-auc:0.792733	validation_1-auc:0.774997
[753]	validation_0-auc:0.792742	validation_1-auc:0.77504
[754]	validation_0-auc:0.792743	validation_1-auc:0.775051
[755]	validation_0-auc:0.792744	validation_1-auc:0.775226
[756]	validation_0-auc:0.792735	validation_1-auc:0.775055
[757]	validation_0-auc:0.792745	validation_1-auc:0.7752
[758]	validation_0-auc:0.792736	validation_1-auc:0.77512
[759]	validation_0-auc:0.792746	validation_1-auc:0.775054
[760]	validation_0-auc:0.79273	validation_1-auc:0.775005
[761]	validation_0-auc:0.792751	validation_1-auc:0.775226
[762]	validation_0-auc:0.792743	validation_1-auc:0.775229
[763]	validation_0-auc:0.792741	validation_1-auc:0.775217
[764]	validation_0-auc:0.79274	validation_1-auc:0.775242
[765]	validation_0-auc:0.79273	validation_1-auc:0.77522
[766]	validation_0-auc:0.792737	validation_1-auc:0.775227
[767]	validation_0-auc:0.792733	validation_1-auc:0.775052
[768]	validation_0-auc:0.792727	validation_1-auc:0.775054
[769]	validation_0-auc:0.792723	validation_1-auc:0.775063
[770]	validation_0-auc:0.792734	validation_1-auc:0.77506
[771]	validation_0-auc:0.792732	validation_1-auc:0.775204
[772]	validation_0-auc:0.792724	validation_1-auc:0.7752
[773]	validation_0-auc:0.792736	validation_1-auc:0.775191
[774]	validation_0-auc:0.792778	validation_1-auc:0.774988
[775]	validation_0-auc:0.792774	validation_1-auc:0.774942
[776]	validation_0-auc:0.792783	validation_1-auc:0.774931
[777]	validation_0-auc:0.792764	validation_1-auc:0.774964
[778]	validation_0-auc:0.792785	validation_1-auc:0.774932
[779]	validation_0-auc:0.792771	validation_1-auc:0.774966
[780]	validation_0-auc:0.792787	validation_1-auc:0.774922
[781]	validation_0-auc:0.792782	validation_1-auc:0.775012
[782]	validation_0-auc:0.792736	validation_1-auc:0.775146
[783]	validation_0-auc:0.792739	validation_1-auc:0.775216
[784]	validation_0-auc:0.792782	validation_1-auc:0.775004
[785]	validation_0-auc:0.792733	validation_1-auc:0.775137
[786]	validation_0-auc:0.792735	validation_1-auc:0.775228
[787]	validation_0-auc:0.792714	validation_1-auc:0.775204
[788]	validation_0-auc:0.792769	validation_1-auc:0.775037
[789]	validation_0-auc:0.792774	validation_1-auc:0.775031
[790]	validation_0-auc:0.792731	validation_1-auc:0.775191
[791]	validation_0-auc:0.792777	validation_1-auc:0.775048
[792]	validation_0-auc:0.792777	validation_1-auc:0.775042
[793]	validation_0-auc:0.792749	validation_1-auc:0.774994
[794]	validation_0-auc:0.792717	validation_1-auc:0.775101
[795]	validation_0-auc:0.792722	validation_1-auc:0.775099
[796]	validation_0-auc:0.792724	validation_1-auc:0.775067
[797]	validation_0-auc:0.792726	validation_1-auc:0.7751
[798]	validation_0-auc:0.792781	validation_1-auc:0.774987
[799]	validation_0-auc:0.792777	validation_1-auc:0.77496
[800]	validation_0-auc:0.79278	validation_1-auc:0.774953
[801]	validation_0-auc:0.792783	validation_1-auc:0.774966
[802]	validation_0-auc:0.792781	validation_1-auc:0.77497
[803]	validation_0-auc:0.792765	validation_1-auc:0.774938
[804]	validation_0-auc:0.79277	validation_1-auc:0.774911
[805]	validation_0-auc:0.792771	validation_1-auc:0.774945
[806]	validation_0-auc:0.792772	validation_1-auc:0.774891
[807]	validation_0-auc:0.792768	validation_1-auc:0.774904
[808]	validation_0-auc:0.792765	validation_1-auc:0.774989
[809]	validation_0-auc:0.792784	validation_1-auc:0.774963
[810]	validation_0-auc:0.792769	validation_1-auc:0.774761
[811]	validation_0-auc:0.792764	validation_1-auc:0.774809
[812]	validation_0-auc:0.792762	validation_1-auc:0.774816
[813]	validation_0-auc:0.79278	validation_1-auc:0.774945
[814]	validation_0-auc:0.79279	validation_1-auc:0.775034
[815]	validation_0-auc:0.792788	validation_1-auc:0.774991
[816]	validation_0-auc:0.79279	validation_1-auc:0.774989
[817]	validation_0-auc:0.792786	validation_1-auc:0.775
[818]	validation_0-auc:0.79278	validation_1-auc:0.774769
[819]	validation_0-auc:0.792792	validation_1-auc:0.774907
[820]	validation_0-auc:0.79275	validation_1-auc:0.775042
[821]	validation_0-auc:0.792747	validation_1-auc:0.775162
[822]	validation_0-auc:0.792743	validation_1-auc:0.775156
[823]	validation_0-auc:0.792742	validation_1-auc:0.775164
[824]	validation_0-auc:0.79274	validation_1-auc:0.77509
[825]	validation_0-auc:0.792747	validation_1-auc:0.775057
[826]	validation_0-auc:0.792746	validation_1-auc:0.775113
[827]	validation_0-auc:0.792747	validation_1-auc:0.775038
[828]	validation_0-auc:0.792747	validation_1-auc:0.775045
[829]	validation_0-auc:0.792738	validation_1-auc:0.774909
[830]	validation_0-auc:0.792737	validation_1-auc:0.775065
[831]	validation_0-auc:0.792718	validation_1-auc:0.775078
[832]	validation_0-auc:0.792792	validation_1-auc:0.774925
[833]	validation_0-auc:0.792783	validation_1-auc:0.77489
[834]	validation_0-auc:0.792784	validation_1-auc:0.774918
[835]	validation_0-auc:0.792788	validation_1-auc:0.774918
[836]	validation_0-auc:0.792738	validation_1-auc:0.775119
[837]	validation_0-auc:0.79278	validation_1-auc:0.77499
[838]	validation_0-auc:0.792734	validation_1-auc:0.775121
[839]	validation_0-auc:0.79274	validation_1-auc:0.775126
[840]	validation_0-auc:0.792747	validation_1-auc:0.775075
[841]	validation_0-auc:0.792745	validation_1-auc:0.775032
[842]	validation_0-auc:0.792741	validation_1-auc:0.775029
[843]	validation_0-auc:0.792738	validation_1-auc:0.775046
[844]	validation_0-auc:0.792731	validation_1-auc:0.774944
[845]	validation_0-auc:0.792722	validation_1-auc:0.77507
[846]	validation_0-auc:0.792748	validation_1-auc:0.775187
[847]	validation_0-auc:0.792746	validation_1-auc:0.775199
[848]	validation_0-auc:0.792745	validation_1-auc:0.775201
[849]	validation_0-auc:0.792744	validation_1-auc:0.775197
[850]	validation_0-auc:0.792746	validation_1-auc:0.775195
[851]	validation_0-auc:0.792796	validation_1-auc:0.774983
[852]	validation_0-auc:0.792794	validation_1-auc:0.775059
[853]	validation_0-auc:0.792783	validation_1-auc:0.775044
[854]	validation_0-auc:0.792772	validation_1-auc:0.775005
[855]	validation_0-auc:0.792779	validation_1-auc:0.775064
[856]	validation_0-auc:0.79279	validation_1-auc:0.775083
[857]	validation_0-auc:0.792791	validation_1-auc:0.775081
[858]	validation_0-auc:0.79279	validation_1-auc:0.77504
[859]	validation_0-auc:0.792794	validation_1-auc:0.774988
[860]	validation_0-auc:0.792794	validation_1-auc:0.774988
[861]	validation_0-auc:0.792792	validation_1-auc:0.774979
[862]	validation_0-auc:0.792749	validation_1-auc:0.775186
[863]	validation_0-auc:0.792741	validation_1-auc:0.775166
[864]	validation_0-auc:0.79278	validation_1-auc:0.77503
[865]	validation_0-auc:0.792788	validation_1-auc:0.775055
[866]	validation_0-auc:0.792746	validation_1-auc:0.775174
[867]	validation_0-auc:0.792723	validation_1-auc:0.775291
[868]	validation_0-auc:0.79275	validation_1-auc:0.775116
[869]	validation_0-auc:0.792751	validation_1-auc:0.775155
[870]	validation_0-auc:0.792748	validation_1-auc:0.775097
[871]	validation_0-auc:0.792747	validation_1-auc:0.775054
[872]	validation_0-auc:0.792749	validation_1-auc:0.775038
[873]	validation_0-auc:0.792738	validation_1-auc:0.775075
[874]	validation_0-auc:0.792741	validation_1-auc:0.775101
[875]	validation_0-auc:0.792736	validation_1-auc:0.775023
[876]	validation_0-auc:0.792741	validation_1-auc:0.775038
[877]	validation_0-auc:0.792743	validation_1-auc:0.775033
[878]	validation_0-auc:0.792726	validation_1-auc:0.775036
[879]	validation_0-auc:0.792729	validation_1-auc:0.775202
[880]	validation_0-auc:0.792735	validation_1-auc:0.775126
[881]	validation_0-auc:0.79273	validation_1-auc:0.775152
[882]	validation_0-auc:0.792785	validation_1-auc:0.775071
[883]	validation_0-auc:0.792783	validation_1-auc:0.774953
[884]	validation_0-auc:0.792776	validation_1-auc:0.774967
[885]	validation_0-auc:0.792776	validation_1-auc:0.774965
[886]	validation_0-auc:0.792789	validation_1-auc:0.774939
[887]	validation_0-auc:0.792783	validation_1-auc:0.774776
[888]	validation_0-auc:0.792787	validation_1-auc:0.774757
[889]	validation_0-auc:0.792786	validation_1-auc:0.774886
[890]	validation_0-auc:0.792788	validation_1-auc:0.77488
[891]	validation_0-auc:0.792789	validation_1-auc:0.774879
[892]	validation_0-auc:0.792785	validation_1-auc:0.774885
[893]	validation_0-auc:0.792734	validation_1-auc:0.775032
[894]	validation_0-auc:0.792788	validation_1-auc:0.774903
[895]	validation_0-auc:0.792746	validation_1-auc:0.775115
[896]	validation_0-auc:0.792786	validation_1-auc:0.774898
[897]	validation_0-auc:0.792739	validation_1-auc:0.775044
[898]	validation_0-auc:0.792782	validation_1-auc:0.774906
[899]	validation_0-auc:0.792783	validation_1-auc:0.775058
[900]	validation_0-auc:0.792789	validation_1-auc:0.775059
[901]	validation_0-auc:0.792788	validation_1-auc:0.775048
[902]	validation_0-auc:0.792792	validation_1-auc:0.775047
[903]	validation_0-auc:0.792793	validation_1-auc:0.775032
[904]	validation_0-auc:0.792788	validation_1-auc:0.775092
[905]	validation_0-auc:0.79279	validation_1-auc:0.775094
[906]	validation_0-auc:0.792797	validation_1-auc:0.775031
[907]	validation_0-auc:0.792797	validation_1-auc:0.775019
[908]	validation_0-auc:0.792774	validation_1-auc:0.774815
[909]	validation_0-auc:0.792781	validation_1-auc:0.77482
[910]	validation_0-auc:0.792797	validation_1-auc:0.775031
[911]	validation_0-auc:0.792795	validation_1-auc:0.775043
[912]	validation_0-auc:0.792786	validation_1-auc:0.775112
[913]	validation_0-auc:0.792766	validation_1-auc:0.775042
[914]	validation_0-auc:0.792756	validation_1-auc:0.775054
[915]	validation_0-auc:0.792768	validation_1-auc:0.775062
[916]	validation_0-auc:0.792779	validation_1-auc:0.774977
[917]	validation_0-auc:0.792788	validation_1-auc:0.775045
[918]	validation_0-auc:0.792766	validation_1-auc:0.774932
[919]	validation_0-auc:0.792791	validation_1-auc:0.774999
[920]	validation_0-auc:0.792771	validation_1-auc:0.774819
[921]	validation_0-auc:0.792771	validation_1-auc:0.774808
[922]	validation_0-auc:0.79276	validation_1-auc:0.774784
[923]	validation_0-auc:0.79276	validation_1-auc:0.774865
[924]	validation_0-auc:0.792758	validation_1-auc:0.774854
[925]	validation_0-auc:0.792703	validation_1-auc:0.774944
[926]	validation_0-auc:0.792714	validation_1-auc:0.774906
[927]	validation_0-auc:0.792781	validation_1-auc:0.774775
[928]	validation_0-auc:0.792734	validation_1-auc:0.774902
[929]	validation_0-auc:0.792741	validation_1-auc:0.774915
[930]	validation_0-auc:0.792737	validation_1-auc:0.774925
[931]	validation_0-auc:0.792784	validation_1-auc:0.774854
[932]	validation_0-auc:0.792742	validation_1-auc:0.775039
[933]	validation_0-auc:0.79276	validation_1-auc:0.774608
[934]	validation_0-auc:0.792713	validation_1-auc:0.774739
[935]	validation_0-auc:0.792784	validation_1-auc:0.774927
[936]	validation_0-auc:0.792739	validation_1-auc:0.775064
[937]	validation_0-auc:0.792739	validation_1-auc:0.775066
[938]	validation_0-auc:0.79278	validation_1-auc:0.774967
[939]	validation_0-auc:0.792779	validation_1-auc:0.774978
[940]	validation_0-auc:0.792758	validation_1-auc:0.774861
[941]	validation_0-auc:0.79278	validation_1-auc:0.774922
[942]	validation_0-auc:0.792784	validation_1-auc:0.774919
[943]	validation_0-auc:0.792784	validation_1-auc:0.774919
[944]	validation_0-auc:0.792759	validation_1-auc:0.774716
[945]	validation_0-auc:0.792774	validation_1-auc:0.774738
[946]	validation_0-auc:0.792787	validation_1-auc:0.774858
[947]	validation_0-auc:0.792782	validation_1-auc:0.774876
[948]	validation_0-auc:0.792784	validation_1-auc:0.774879
[949]	validation_0-auc:0.792782	validation_1-auc:0.774864
[950]	validation_0-auc:0.792781	validation_1-auc:0.77492
[951]	validation_0-auc:0.792779	validation_1-auc:0.774943
[952]	validation_0-auc:0.792777	validation_1-auc:0.774866
[953]	validation_0-auc:0.792778	validation_1-auc:0.774863
[954]	validation_0-auc:0.792778	validation_1-auc:0.774893
[955]	validation_0-auc:0.792757	validation_1-auc:0.774684
[956]	validation_0-auc:0.792756	validation_1-auc:0.774677
[957]	validation_0-auc:0.792751	validation_1-auc:0.774702
[958]	validation_0-auc:0.79273	validation_1-auc:0.774604
[959]	validation_0-auc:0.792732	validation_1-auc:0.774564
[960]	validation_0-auc:0.792756	validation_1-auc:0.774702
[961]	validation_0-auc:0.79275	validation_1-auc:0.774698
[962]	validation_0-auc:0.792721	validation_1-auc:0.77481
[963]	validation_0-auc:0.792742	validation_1-auc:0.774489
[964]	validation_0-auc:0.792761	validation_1-auc:0.774682
[965]	validation_0-auc:0.792754	validation_1-auc:0.774619
[966]	validation_0-auc:0.792735	validation_1-auc:0.774454
[967]	validation_0-auc:0.792731	validation_1-auc:0.774435
[968]	validation_0-auc:0.792727	validation_1-auc:0.774461
[969]	validation_0-auc:0.792755	validation_1-auc:0.774526
[970]	validation_0-auc:0.792757	validation_1-auc:0.774614
[971]	validation_0-auc:0.79276	validation_1-auc:0.774613
[972]	validation_0-auc:0.792772	validation_1-auc:0.774757
[973]	validation_0-auc:0.792763	validation_1-auc:0.774656
[974]	validation_0-auc:0.792761	validation_1-auc:0.774717
[975]	validation_0-auc:0.792761	validation_1-auc:0.774655
[976]	validation_0-auc:0.792761	validation_1-auc:0.774655
[977]	validation_0-auc:0.792757	validation_1-auc:0.774636
[978]	validation_0-auc:0.792708	validation_1-auc:0.774682
[979]	validation_0-auc:0.792708	validation_1-auc:0.774677
[980]	validation_0-auc:0.792708	validation_1-auc:0.774651
[981]	validation_0-auc:0.792728	validation_1-auc:0.774865
[982]	validation_0-auc:0.792728	validation_1-auc:0.774865
[983]	validation_0-auc:0.792729	validation_1-auc:0.774877
[984]	validation_0-auc:0.792781	validation_1-auc:0.774839
[985]	validation_0-auc:0.792741	validation_1-auc:0.774998
[986]	validation_0-auc:0.792741	validation_1-auc:0.774998
[987]	validation_0-auc:0.792788	validation_1-auc:0.774925
[988]	validation_0-auc:0.792777	validation_1-auc:0.774904
[989]	validation_0-auc:0.792748	validation_1-auc:0.774901
[990]	validation_0-auc:0.79277	validation_1-auc:0.774816
[991]	validation_0-auc:0.792783	validation_1-auc:0.775031
[992]	validation_0-auc:0.792765	validation_1-auc:0.775016
[993]	validation_0-auc:0.792735	validation_1-auc:0.775167
[994]	validation_0-auc:0.792771	validation_1-auc:0.774996
[995]	validation_0-auc:0.792769	validation_1-auc:0.774994
[996]	validation_0-auc:0.792786	validation_1-auc:0.774849
[997]	validation_0-auc:0.792791	validation_1-auc:0.774998
[998]	validation_0-auc:0.792785	validation_1-auc:0.774871
[999]	validation_0-auc:0.792784	validation_1-auc:0.774879
Wall time: 10.8 s
Out[107]:
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1,
              disable_default_eval_metric=1, eval_metric='auc', gamma=0.15,
              learning_rate=0.1, max_delta_step=0, max_depth=7,
              min_child_weight=3, missing=None, n_estimators=1000, n_jobs=8,
              nthread=None, objective='binary:logistic', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=7.8243170862346005,
              seed=None, silent=None, subsample=0.5, verbose_eval=100,
              verbosity=0)
In [108]:
true_flag = test_df_total['y']
pred_flag = clf4.predict(test_df_total[top_5_feat_2])

print("AUC score: {:<8.3f}".format(roc_auc_score(true_flag, pred_flag)))
AUC score: 0.725   

Check confusion matrix, AUC plot and other metrics

In [109]:
metrics = confusion_mat_plot(true_flag, pred_flag)
accuracy: 83.07 %
precision: 34.3 %
Recall: 58.94 %
specificity: 86.05 %
F-score: 43.36 %
In [110]:
gini = auc_plot(test_df_total[top_5_feat_2], true_flag, clf4)

# record new metrics
model_comparison['Without Eng Feat-Top 5'] = [gini, metrics[0], metrics[1], metrics[2], metrics[3], metrics[4]]
gini 54.976 %

We observe again a small sacrifice in performance, however, we have a less complicated model.

Let's check again the potential changes in feature importance (without engineered features).

In [111]:
feature_importance_df4 = pd.DataFrame()
feature_importance_df4["Feature"] = top_5_feat_2
feature_importance_df4["importance"] = clf4.feature_importances_

# plot results
plt.figure(figsize=(12,12))
sns.barplot(x="importance", y="Feature", data=feature_importance_df4.sort_values(by="importance",ascending=False))
plt.title('Features importance')
plt.tight_layout()
plt.show()

Conclusion

As we can see, the most predictive features that define the model's decision are the rr.employed and the result of the previous campaign, the feature previous_success.

There is a lot more analysis that can be done to find the best-tuned model for the topic. Other areas that we could touch to improve the model are:

  • PCA analysis of the variables.
  • We could apply Explainable AI libraries like SHAP and LIME and have a better understanding of what drives the decision on each occasion in a tree-based model, especially for the successfully subscribed customers.
  • Due to the imbalanced nature of the dataset, we could actually apply techniques like SMOTE, to synthesize more balanced datasets that could potentially boost the training performance of our models.
  • Eventually, if we are satisfied with the accuracy, we can try and apply some linear models that would be expected to be less performant, but more explainable like logistic regression.

Below we can see the cummulative results of all models:

Gini Performance of All Models

In [112]:
%%time
# create all model arguments
X_test = [test_df_total[features], test_df_total[top_5_feat],
          test_df_total[train_df_total.columns[:-9]], test_df_total[top_5_feat_2]]
y_test = [true_flag, true_flag, true_flag, true_flag]
clf = [clf1, clf2, clf3, clf4]
model_names = model_comparison.columns.to_list()
model_weights = [None, None, None, None]
Wall time: 4 ms
In [113]:
%time
auc_plot_all(X_test, y_test, clf, model_names, w=model_weights)
Wall time: 0 ns

Performance Metrics for All Models in 50% Cut-off point

In [114]:
%%time
plot_models_radar(model_comparison)
Wall time: 197 ms

Model Comparison on Different Cut-offs (interactive slider)

In [115]:
%%time
metrics_slider_plot(X_test, y_test, clf, model_names)
Wall time: 14.7 s